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ABSTRACT 


In  the  field  of  human  research,  particularly  in  operational  environments,  data 
collection  techniques  are  difficult.  Researchers  often  focus  their  efforts  on  the  data 
analysis  and  overlook  the  shortcomings  of  their  data  collection  and  storage 
methodologies.  In  order  to  demonstrate  effective  data  collection  and  storage 
methodology  in  a  representative  human  research  process,  the  process  used  by  human 
fatigue  and  performance  researchers  at  the  Human  Systems  Integration  Lab  at  Naval 
Postgraduate  School  (NPS)  served  as  a  Proof  of  Concept.  Most  recent  studies  conducted 
at  NPS  provided  a  model  of  the  current  process.  The  Knowledge  Value  Added  (KVA) 
methodology  was  used  as  a  tool  of  comparison  of  the  current  process  to  the  reengineered 
process.  Information  technologies  including  wireless  physiological  monitoring  devices, 
web-based  and  mobile  device  data  collection  methods,  and  integrated  data  storage 
techniques  were  incorporated  in  the  reengineering  effort.  The  data  storage  process 
included  the  design  of  a  standard  relational  database  format  allowing  research  teams  to 
easily  access  their  data.  This  repository  also  enables  data  to  be  archived  for  future  use 
(e.g.,  meta-analyses).  To  demonstrate  the  reengineered  process  in  an  operational 
environment,  a  field  fatigue  study  was  conducted  at  the  Naval  Officer  Indoctrination 
School  (OIS)  in  Newport,  Rhode  Island. 
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1.  INTRODUCTION 


A.  OVERVIEW 

Data  collection  in  the  field  of  human  research  is  intrinsically  difficult. 
Monitoring  physical  and  psychological  behaviors  through  various  data  collection  media 
and  integrating  the  collected  data  points  comprehensively  for  effective  and  timely 
analysis  is  a  challenge  in  a  carefully  controlled  laboratory  environment.  However,  when 
the  collection  takes  place  in  operational  environments,  the  complexity  intensifies.  Often, 
the  research  team  becomes  focused  on  data  analysis  and  they  overlook  major 
shortcomings  in  their  collection  and  storage  methodologies.  Collecting  human 
performance  and  fatigue  data  are  one  of  the  most  complex  research  areas  in  the  field  of 
human  research.  In  order  to  demonstrate  effective  data  collection  and  storage 
methodology  in  a  representative  human  research  process,  the  process  used  by  fatigue 
researchers  at  the  Human  Systems  Integration  Lab  at  Naval  Postgraduate  School  (NPS), 
Monterey,  California,  served  as  a  Proof  of  Concept.  Various  information  technologies 
were  considered  in  efforts  to  streamline  the  data  collection  and  storage  processes  to 
improve  the  productivity  of  human  research  conducted  under  operational  conditions. 

This  process  used  in  recent  studies  conducted  by  faculty  and  students  at  NPS  of 
collecting  and  analyzing  human  performance  and  fatigue  data  were  evaluated  and 
reengineered.  Web-based  data  collection  methods  and  mobile  communications  devices 
were  incorporated  in  the  redesigned  data  collection  sub -process  enabling  additional  data 
points  to  be  collected  in  a  more  efficient  and  timely  manner  with  greater  accuracy.  The 
data  storage  process  was  redesigned  as  well  by  establishing  a  standard  relational  database 
format  allowing  research  teams  to  easily  access  their  data.  This  repository  also  enables 
data  to  be  archived  for  future  use  (e.g.,  meta-analyses).  A  study  was  conducted  at  the 
Naval  Officer  Indoctrination  School  (OIS)  in  Newport,  Rhode  Island  to  demonstrate  the 
newly  optimized  process. 

B.  BACKGROUND 

To  improve  the  real  life  human  performance,  human  research  must  be  conducted 
in  operational  settings.  Because  of  the  difficult  challenges  environmental  conditions 

place  on  such  research,  it  is  even  more  important  for  researchers  to  collect  data  efficiently 
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and  as  unobtrusively  as  possible.  There  is  often  only  one  chance  for  chta  collection. 
Since  so  much  is  at  stake  under  these  conditions,  it  is  essential  that  perhaps  the  only 
chance  to  capture  human  statistics  under  such  circumstances  not  be  lost  due  to  poor  data 
collection  methodology.  Of  all  of  the  areas  of  human  research,  fatigue  as  it  relates  to 
human  performance  is  particularly  complex.  Additionally,  it  has  become  of  increasing 
concern  for  the  military  as  the  force  has  been  tasked  to  accomplish  more  with  less  people. 

Fatigue  management  has  become  a  significant  issue  in  effective  warfighting.  The 
news  media,  embedded  with  the  forces  on  the  ground  and  at  sea  during  the  war  in  Iraq, 
continuously  reported  the  sleep  deprived  state  of  the  men  and  women  serving  in  the 
armed  forces  and  the  potential  dangers  fatigue  had  on  their  vigilance  while  operating 
technical  systems.  The  awareness  of  the  consequences  of  fatigue  continues  to  grow.  As 
this  occurs,  a  desire  for  innovative  solutions  to  mitigate  the  given  combat  circumstances 
and  achieve  maximum  human  performance  is  at  a  record  high. 

At  NFS,  the  current  process  of  collecting  fatigue  data  in  an  operational 
environment,  analyzing  the  data,  and  publishing  the  results  takes  about  a  year  to 
complete.  The  researchers  for  this  process  are  thesis  students  who  require  significant 
training  in  the  science  of  conducting  human  research  as  well  as  how  to  use  the  collection 
equipment  and  analysis  tools  available  to  them  at  NFS.  In  order  to  arm  future  sleep 
research  teams  with  the  ability  to  identify  sponsoring  organization’s  fatigue  issues  in  a 
timely  manner,  the  current  fatigue  study  process  has  to  change.  Using  this  current 
process,  identifying  potential  danger  areas  is  the  extent  of  the  analysis.  There  is  little 
time  available  to  propose  innovative  solutions  for  the  sponsoring  organization.  The  new 
process  must  reduce  the  time  required  to  conduct  the  study  and  provide  for  collection  of 
enough  quality  data  to  enable  the  team  to  draw  conclusions  and  recommend  solutions  or 
at  the  least,  identify  areas  to  focus  further  study  if  required.  Additionally,  a  capability  to 
enable  meta- analysis  of  many  studies  conducted  by  NFS  students  and  faculty  would 
clearly  provide  a  powerful  tool  not  currently  available  as  data  are  stored  in  disparate 
structures  using  many  unique  systems. 
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C.  RESEARCH  QUESTIONS 

•  How  much  can  the  productivity  of  the  current  process  of  data  collection  of 
human  performance  and  fatigue  data  during  studies  conducted  at  the  NPS 
be  improved  through  redesign  using  information  technology? 

•  How  will  web-based  technology  facilitate  data  collection  in  the  process 
redesign? 

•  How  will  modern  database  technology  enable  efficiently  structured  data 
storage  in  the  process  redesign? 

•  How  will  current  mobile  personal  computing  device  (PDA)  technology 
facilitate  data  collection  and  integration  in  the  process  redesign? 

•  How  will  performance  testing  using  a  PDA  aid  the  process 
redesign? 

•  How  will  development  of  a  PDA  based  tracking  log  give  support 
to  process  redesign? 

D.  SCOPE  OF  THESIS 

In  efforts  to  improve  data  collection  and  storage  processes  in  the  field  of  human 
research,  the  current  fatigue  study  process  at  NPS  was  examined  as  the  Proof  of  Concept. 
The  process  was  analyzed,  evaluated,  and  reengineered  using  information  technology  to 
provide  an  optimal  solution.  The  most  significant  areas  of  improvement  included  the 
innovative  data  collection  and  data  storage  methods.  The  optimized  model  was  tested  as 
data  were  collected,  analyzed,  and  stored.  After  necessary  revisions  to  the  model,  an 
operational  study  was  conducted  at  the  Navy  OIS  in  Newport,  Rhode  Island  to  validate 
the  reengineered  process.  The  scope  is  limited  to  the  collection  and  storage  of  data. 
Research  efforts  toward  interpretation  of  the  data  collected  during  this  study  were 
focused  to  validate  the  reengineered  process  rather  than  identify  issues  of  fatigue  of  the 
study  participants. 

E.  RESEARCH  METHODOLOGY 

To  determine  the  best  approach  to  reengineer  the  process,  the  current  methods  of 
collection  and  analysis  were  studied  and  the  Knowledge  Value  Added  (KVA) 
methodology  was  used.  KVA  provides  a  way  to  capture  knowledge  within  a  set  of 
processes  and  measure  its  associated  value.  The  KVA  methodology  provided  the  ability 
to  evaluate  the  current  process  and  establish  a  baseline  for  comparison  to  the 
reengineered  process.  The  redesigned  process  was  developed  using  a  three  tiered 
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architecture  approach  and  was  evaluated  using  the  KVA  method.  Finally,  this  process 
was  tested  in  the  Human  Systems  Integration  Lab  at  NFS,  fine-tuned,  and  was  ultimately 
field  tested  at  the  Navy  OIS  in  Newport,  Rhode  Island. 

F.  THESIS  ORGANIZATION 

Chapter  II  follows  with  a  Literature  Review  including  an  overview  of  human 
performance  and  effects  of  fatigue  as  well  as  a  description  of  the  software  tools  used 
during  fatigue  studies  for  data  collection  and  analysis.  The  current  process  of  conducting 
these  studies  is  outlined  in  Chapter  III.  An  explanation  of  the  process  model  and  an 
evaluation  using  KVA  analysis  is  provided.  Chapter  IV  summarizes  the  technical 
approach  used  in  reengineering  the  current  process  to  include  the  parameters  of  design 
and  the  data  collection  and  storage  methods  considered.  Chapter  V  provides  the  model 
for  the  reengineered  process  and  an  evaluation  of  it  using  the  KVA  methodology.  The 
validation  of  the  new  process  is  explained  in  Chapter  VI  with  a  description  of  the  study 
conducted  with  the  OIS  participants  in  Newport,  Hiode  Island.  Finally,  Chapter  VII 
concludes  with  recommendations  for  future  work. 
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n.  LITERATURE  REVIEW 


A.  OVERVIEW 

This  literature  review  covers  several  diverse  areas.  To  assist  in  evaluation  of  the 
data  collection  and  storage  processes  of  the  Proof  of  Concept,  this  review  is  meant  to 
provide  the  reader  with  an  overall  understanding  of  the  field  of  sleep,  fatigue  and  human 
performance  and  relevant  research  techniques.  In  order  to  adequately  assess  the  results  of 
the  reengineered  process,  an  introduction  to  the  Knowledge  Value  Added  methodology  is 
also  presented.  Though  the  focus  of  this  review  is  on  the  data  collection  techniques 
specific  to  the  fatigue  research  process,  the  methods  of  data  collection  presented  are 
intended  to  serve  as  an  example  of  the  numerous  means  of  data  collection  used  in  human 
research. 

The  effects  of  fatigue  and  sleep  deprivation  on  human  performance  have  been  of 
keen  interest  to  those  in  the  field  of  sleep  research  and  those  in  the  business  of  optimizing 
human  performance  effectiveness.  This  review  describes  human  physiology  and  the  role 
sleep  plays  in  biological  systems.  The  impact  of  sleep  deprivation  on  human  physiology 
and  performance  is  introduced  as  well  as  reports  of  findings  in  recent  dose-response 
studies.  Additionally,  current  methods  of  data  collection  and  analysis  used  to  document 
fatigue  as  it  relates  to  human  performance  are  described. 

As  the  military  continues  to  require  warfighters  to  operate  under  conditions  of 
chronic  sleep  deprivation,  the  risks  have  become  clear.  Recent  studies  conducted  by 
students  and  faculty  at  NFS  highlight  the  military’s  continued  interest  in  documenting  the 
effects  of  fatigue  on  human  performance  in  operational  environments.  These  efforts 
provide  the  necessary  evidence  to  implement  changes  enabling  mitigation  of  risks 
imposed  by  lack  of  sleep.  Following  these  descriptions,  the  current  tools  used  during 
sleep  studies  by  NFS  research  teams  are  introduced.  This  section  discusses  data 
collection  tools  as  well  as  the  software  required  to  download  and  analyze  fatigue  data. 
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In  the  final  section  of  this  chapter,  an  overview  of  the  Knowledge  Value  Added 
approach  to  assessing  the  value  of  knowledge  in  a  given  process  will  be  introduced.  This 
methodology  provides  the  means  to  compare  the  current  process  used  by  these  recent 
NPS  studies  to  the  reengineered  process  proposed  from  this  research. 

B.  HUMAN  PERFORMANCE  AND  FATIGUE 

1.  Human  Physiology  and  the  Role  of  Sleep 

Human  beings,  evolving  over  time,  have  developed  an  internal  biological  clock 
that  serves  to  regulate  sleep  and  wake  periods.  These  endogenous  clocks  have  been 
described  to  control  a  number  of  biochemical,  physiological,  and  behavioral  phenomena 
in  humans.  Operating  on  a  cyclic  pattern,  these  clocks  repeat  approximately  once  per  day 
and  are  called  circadian  rhythms.  Franz  Halberg  of  the  University  of  Minnesota  first 
used  the  word  circadian  as  it  comes  from  the  Latin  ‘circa’  (about)  and  ‘dies’  (day).  In  the 
late  1960s,  scientists,  including  Jurgen  Acshoff,  at  the  Max  Planck  Institute  in  Germany 
isolated  individuals  from  any  external  environmental  cues  and  allowed  the  human 
participants  to  set  their  own  schedules.  After  days  of  isolation,  participants  adjusted  to  a 
regular  cycle  of  sleeping  and  waking.  These  scientists  also  discovered  that  the  internal 
clock  cycle  repeated  approximately  every  25  hours  (Dement,  1999). 

Because  the  25  hour  cycle  time  varies  slightly  from  the  Earth’s  rotation  cycle,  it  is 
necessary  for  humans  to  receive  environmental  time  cues  called  ‘zeitgebers’  which  can 
help  to  reset  the  biological  clock.  The  location  of  the  internal  clock  appears  to  be  in  a 
small  area  of  the  hypothalamus  known  as  the  suprachiasmatic  nucleus  (SCN).  It  is 
sensitive  to  the  inputs  from  the  optic  nerve.  These  inputs  act  as  a  controlling  mechanism 
making  necessary  adjustments  to  the  internal  clock.  Because  of  this,  light  is  the  most 
powerful  zeitgeber  for  humans.  Charles  A.  Czeisler  performed  several  studies  to 
demonstrate  light’s  influence  on  circadian  rhythms.  His  results  conclude  that  even  dim 
light  affects  this  circadian  clock  (Czeisler,  1990).  Figure  1  shows  the  pattern  for  typical 
alertness  over  the  course  of  a  day.  The  effects  of  accumulated  sleep  dept  as  a  result  of 
extended  wakefulness  counter  the  effects  of  the  biological  clock.  Drs.  Charles  Edgar  and 
William  Dement  defined  an  opponent-process  model  which  suggests  that  the  alerting 
mechanism  of  the  biological  clock  as  well  as  a  sleep  drive  that  increases  during  waking 
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hours  as  sleep  debt  accumulates  are  the  two  processes  that  regulates  sleep  in  humans. 
The  alerting  mechanisms  occur  when  a  person  awakens  in  the  morning  and  in  the  late 
afternoon  (Edgar  et.  al,  1993). 
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Figure  1.  Alertness  Over  24-Hour  Period  (From:  Maas,  1998) 

2.  Performance  Degradation  Due  to  Fatigue 

As  neurological  studies  continue  to  examine  what  the  brain  and  other  body 
functions  do  during  sleep,  the  exact  activities  remain  largely  unknown.  Progress  has 
been  made  in  assessing  the  effects  that  sleep  deprivation  has  on  performance. 
Researchers  agree  undisputedly  that  total  sleep  restriction  results  in  huge  performance 
decrements.  The  extent  of  the  performance  degradation  with  partial  sleep  deprivation  is 
currently  the  focus  of  much  research.  When  comparing  the  effects  of  chronic  partial 
sleep  deprivation  to  the  effects  of  complete  sleep  restriction,  the  resulting  performance 
decrement  is  surprisingly  similar.  In  a  recent  dose-response  study  conducted  at  Walter 
Reed  Army  Institute  of  Research,  individuals  were  allowed  to  sleep  for  a  restricted 
amount  of  time  over  a  seven  day  period.  Sixty-six  people  spent  either  3,  5,  7,  or  9  hours 
in  bed  during  this  period  (Belenky,  et.  al,  2003).  Over  the  course  of  the  study,  the 
participants  conducted  psychomotor  vigilance  tests  to  measure  their  simple  reaction  time. 
The  participants  pushed  a  thumb  operated  handheld  device  as  a  reaction  to  a  visual 
stimulus.  The  results  of  their  psychomotor  vigilance  testing  (PVT)  can  be  seen  in  Figure 
2.  As  expected,  PVT  speeds  decrease  over  time  with  the  most  dramatic  changes  seen  in 
the  group  who  spent  three  hours  a  night  in  bed.  All  of  the  participants  were  given  a  three 
day  recovery  period  where  there  were  no  restrictions  on  sleep.  After  three  days,  none  of 
the  sleep  deprived  groups  returned  to  their  baseline  performance  speed.  The  conclusions 
drawn  by  the  researchers  suggest  that  the  human  body,  in  a  state  of  chronic  deprivation. 
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adapts  by  stabilizing  the  individual’s  performance  to  a  level  below  that  of  the  initial 
baseline.  This  stabilization,  however,  prevents  the  rapid  recovery  to  baseline 
performance.  This  finding  is  in  contrast  to  the  rapid  recovery  in  performance  seen  in 
individuals  whose  sleep  has  been  completely  restricted. 


Figure  2.  Mean  Psychomotor  Vigilance  Task  Speed  (and  Standard  Error) 
Across  Days  as  a  Function  of  Time  (From:  Belenky,  et  al,  2003) 

A  similar  study  conducted  at  the  University  of  Pennsylvania  restricted  participants 
to  either  4,  6,  or  8  hours  of  sleep  per  night  for  a  period  of  14  days.  Another  group  was 
kept  awake  for  88  hours.  All  participants  were  given  a  performance  assessment  battery. 
This  battery  included  a  psychomotor  vigilance  test  similar  to  the  one  used  in  the  Walter 
Reed  study  to  measure  behavioral  alertness  as  well  as  a  computerized  digit  symbol 
substitution  test  to  measure  working  memory.  Researchers  found  that  subjects  whose 
sleep  period  was  4  to  6  hours  a  night  had  significant  deficits  in  cognitive  performance, 
although  the  subjects  reported  being  only  slightly  sleepy.  Those  participants  who  were 
completely  deprived  of  sleep  had  cognitive  deficiencies  similar  to  the  reduced  sleep 
group;  however,  they  reported  being  very  tired.  The  results  indicate  that  although  partial 
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sleep  deprivation  significantly  decreases  performance,  individuals  are  unaware  of  this 
decrease  in  efficiency  (Van  Dongen,  et.  al,  2003).  These  study  results  present  a  clearer 
picture  of  possible  dangers  to  members  of  the  military  as  well  as  medical  professionals 
and  shift  workers.  In  occupations  where  6  hours  of  sleep  a  night  or  less  is  the  norm, 
individuals  may  be  operating  at  decreased  performance  fevels  without  even  knowing  it. 
The  potential  risks  associated  with  the  errors  made  when  performance  degrades  are 
alarming. 

3.  Data  Collection  Techniques  (Physiology  and  Performance) 

Human  research  includes  the  collection  of  both  physiological  and  psychological 
data  points.  The  field  of  biometrics  has  expanded  so  much  recently  that  individuals 
conducting  human  research  or  simply  gathering  human  physiological  data  have  many 
wireless  devices  available  today  for  data  collection.  From  portable  heart  monitors  to 
caloric  intake  measurement  devices,  innovative  measurement  techniques  are  allowing 
research  to  be  conducted  in  much  more  realistic  environments.  The  explosion  of  wireless 
technology  in  computing  allows  PDAs  and  other  handheld  devices  to  enable  the 
collection  of  an  endless  assortment  of  data  including  but  not  limited  to  survey  data  and 
human  performance  data  with  minimal  disruption  to  the  individual  participating  in  the 
research.  As  examples  of  some  of  the  more  intricate  methods  of  data  collection  involving 
humans,  fatigue  and  human  performance  measurement  techniques  provide  challenges  to 
field  researchers. 

Assessments  of  fatigue  and  human  performance  in  particular  have  typically  been 
accomplished  within  the  laboratory  setting  using  a  variety  of  standard  measurement  tools. 
Sleep  researchers  frequently  use  electrophysiology  to  assess  sleep.  Human  adults 
experience  five  stages  of  sleep  during  a  typical  night’s  sleep:  stages  one  through  four  and 
REM  (Rapid  Eye  Movement)  sleep.  Stages  one  through  four  are  referred  to  as  Non- REM 
(NREM)  sleep.  Each  sleep  stage  has  its  own  identifying  marks  as  measured  by 
electroencephalographs  (EEGs),  electromyograms  (EMGs),  and  electrooculograms 
(EOGs).  EEGs  measure  brain  waves  via  electrodes  pasted  to  the  scalp.  EMGs  measure 
muscle  activity  and  EOGs  measure  eye  movements  also  using  contact  electrodes.  The 
collective  group  of  EEG,  EMG  and  EOG  measurements  taken  when  studying  sleep  stages 
is  referred  to  as  Polysomnography  (PSG).  The  physiologic  sensor  leads  are  placed  on  the 
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participant  in  order  to  record  any  number  of  the  following  activities:  brain  electrical 
activity,  eye  and  jaw  muscle  movement,  leg  muscle  movement,  airflow,  respiratory 
effort,  and  oxygen  saturation.  Although  these  measurement  techniques  have  proven  to  be 
very  effective,  they  are  highly  sophisticated,  very  expensive,  and  extremely  obtrusive  to 
the  individual. 

When  collecting  data  on  participants  in  their  work  environments  as  the  field 
studies  conducted  by  NPS  research  teams  do,  less  obtrusive  means  of  measuring  sleep  are 
required.  The  use  of  actigraphy  to  measure  movement  has  become  the  most  effective 
means  for  researchers  to  capture  sleep/wake  periods  of  study  participants  in  field  studies. 
Many  types  of  small  wireless  monitoring  devices  are  available  on  the  market  today  that 
provide  the  capability  to  measure  a  number  of  physiological  activities.  Wrist  activity 
monitors  or  WAMs  are  the  size  of  a  wrist  watch  and  contain  an  accelerometer  that 
measures  movement.  The  associated  software  of  the  WAMs  use  algorithms  to  calculate 
sleep  and  wake  periods  based  on  measured  activity  levels.  WAMs,  however,  do  not 
allow  the  researcher  to  measure  brain  activity.  Therefore,  in  operational  studies  where 
WAMs  are  used,  researchers  cannot  measure  the  stages  of  sleep  participants  reach  but 
only  can  determine  when  the  participants  slept  and  when  they  were  awake.  Sleep 
efficiency  can  be  calculated  from  the  WAM  data  as  a  ratio  of  sleep  epochs  to  actual  sleep 
period.  Sleep  latency  can  also  be  calculated  from  the  time  an  individual  goes  down  for 
sleep  until  the  time  he  is  actually  asleep.  The  various  WAMs  on  the  market  today  use 
different  methods  to  record  movement  and  offer  different  software  tools  to  provide 
unique  analytical  techniques. 

Collecting  additional  variables  such  as  melatonin  levels  or  ambient  light  exposure 
are  useful  techniques  used  in  predicting  circadian  rhythms.  Researchers  measure 
hormone  levels  such  as  melatonin  by  collecting  saliva  or  blood  samples  which  have  a 
distinctive  circadian  pattern.  Other  measurements  such  as  light  exposure,  body 
temperature  or  subjective  activity  logs  are  useful  for  assisting  researchers  in 
understanding  circadian  patterns  as  well  as  interpreting  individual  behavior  and 
predicting  effectiveness. 
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Various  techniques  have  been  developed  to  measure  performance  or  model 
predicted  effectiveness.  Cognitive  testing  is  one  approach  to  measuring  effectiveness. 
Psychomotor  vigilance  tests  including  reaction  time  tests,  running  memory  tests,  math 
processing  tests,  and  others  are  used  by  researchers  to  measure  performance.  One  of  the 
tools  created  specifically  for  field  environments  is  the  ARES  (ANAM^'^  Readiness 
Evaluation  System).  Included  in  this  tool  is  the  Stanford  Subjective  Sleepiness  Scale  that 
identifies  the  particular  sleepiness  state  felt  by  the  participants  a  given  time.  One  tool 
used  for  predicting  cognitive  effectiveness  is  the  Eatigue  Avoidance  Scheduling  Tool 
(EAST)  built  on  principles  of  the  Sleep,  Activity,  Eatigue,  and  Task  Effectiveness 
(SAETE)  Model.  Analysis  of  data  collected  on  participants  engaged  in  the  performance 
of  cognitive  tasks  during  several  sleep  deprivation  studies  conducted  by  the  Army,  Air 
Eorce  and  Canadian  researchers  is  the  basis  for  the  expected  level  of  performance 
effectiveness  predicted  by  the  EAST.  Though  designed  as  a  scheduling  tool,  the  EAST 
can  serve  as  a  model  to  determine  effectiveness  levels  based  on  collected  WAM 
accelerometer  data. 

C.  RECENT  STUDIES  CONDUCTED  AT  THE  NAVAU  POSTGRADUATE 

SCHOOU 

As  recent  studies  continue  to  show  the  strong  correlation  between  sleep 
deprivation  and  the  degradation  of  human  performance,  the  United  States  military  has 
developed  a  keen  interest  in  studying  effects  of  fatigue  on  service  members.  Students 
and  faculty  at  NPS  have  recently  conducted  studies  to  assess  these  effects  in  various 
operational  environments  in  the  Navy. 

ET  John  Nguyen  collected  data  on  the  aircraft  carrier  the  USS  JOHN  C. 
STENNIS  during  a  shift  from  day  operations  to  night  operations  in  support  of 
OPERATION  ENDURING  EREEDOM.  He  used  Actigraphs  to  collect  actigraphy  and 
sleep  data  and  required  the  participants  to  keep  a  daily  journal  of  their  sleep  activity.  He 
also  collected  oral  temperatures  of  the  participants  every  three  waking  hours  and  required 
each  participant  to  fill  out  several  surveys  to  support  his  research.  His  thesis  revealed  the 
difficulty  experienced  by  sailors  when  reversing  sleep  schedules  from  day  to  night 
operations.  Additionally,  ET  Nguyen’s  data  exposed  significant  differences  between 
those  sailors  working  topside  and  those  working  below  decks.  As  a  result  of  his  findings. 


11 


interest  has  been  generated  to  more  closely  understand  the  effects  that  ambient  light  and 
other  zeitgebers  have  on  sailors  in  operational  situations.  While  it  is  understood  that 
mission  demands  may  not  allow  optimal  sleep  schedules  and  sterile  environments  to  be 
created  and  applied  on  deployed  carriers,  this  study  concluded  that  investigations  into 
optimizing  schedules  and  environments  within  the  constraints  of  a  given  deployment  may 
be  worth  the  effort.  Potentially  maximizing  human  performance  effectiveness  as  it 
relates  to  sleep  deprivation  in  the  arduous  environments  they  face  at  war  is  the  possibility 
highlighted  in  LT  Nguyen’s  research. 

During  another  study  conducted  at  the  Great  Lakes  Naval  Recruiting  Station,  LT 
Brian  Baldus  examined  the  effects  of  changing  bedtime  for  Navy  recruits  from  2100  to 
2200.  Using  the  WAMs  and  various  subjective  tests,  he  concluded  that  the  change  in 
time  was  significant  between  the  two  sleep  condiions  (Baldus,  2002).  He  concluded  that 
recruits  who  were  able  to  go  to  bed  later  were  able  to  get  to  sleep  faster  and  therefore 
sleep  longer.  This  increase  ultimately  improved  their  performance  effectiveness. 
Because  the  average  ages  of  the  study  population  are  adolescents  and  young  adults,  the 
later  bedtime  aligned  more  appropriately  with  the  known  differences  in  adolescent/young 
adult  circadian  rhythms.  Other  training  institutions  throughout  the  military  have  shown 
interest  in  these  results,  prompting  the  desire  for  additional  studies  to  find  optimal  sleep 
plans  for  unique  circumstances  in  training  as  well  as  in  operational  environments. 

Two  other  recent  NPS  theses  examine  sleep  in  operational  settings  (Blassingame, 
2001  and  Gamboa,  2002).  In  an  analysis  of  archival  submarine  survey  data  collected  by 
the  Naval  Submarine  Medical  Research  Laboratory  (NSMRL),  LT  Donnie  Gamboa  and 
LT  Simonia  Blassingame  revealed  that  sailors  have  shown  characteristics  of  sleep 
deprivation  that  they  claim  inhibit  their  ability  to  stand  watch  at  an  optimum  level.  His 
research  focused  on  answering  questions  related  to  environmental  constraints  and  time  in 
service  as  factors  related  to  sleep  and  fatigue.  LT  Gamboa  also  presented  a  useful 
discussion  regarding  the  importance  of  sleep  management  in  optimizing  operational 
performance.  His  conclusions  emphasize  the  requirement  to  manage  sleep  as  a  resource 
for  warfighters.  He  says  the  potential  for  mismanaged  sleep  could  most  certainly  result 
in  missions  being  compromised  (Gamboa,  2002). 
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D.  MEASUREMENT  TOOLS  AND  SOFTWARE  USED  IN  NFS  STUDIES 

The  specific  measurement  techniques  and  tools  used  in  recent  NFS  fatigue  and 
performance  studies  are  described  in  the  following  paragraphs.  The  particular  WAM 
device  used  is  called  an  Actigraph.  It  was  developed  by  the  Precision  Control  Design, 
Inc.  (PDC).  The  sleep  watches,  along  with  accompanying  software  to  initialize  and 
download  the  data,  are  marketed  by  Ambulatory  Monitoring,  Inc.  (AMI).  Dr.  Timothy 
Elsmore  from  Activity  Research  Services,  Inc.  has  authored  the  Action-W  software 
(marketed  by  AMI)  which  is  a  Windows-based  program  used  for  analysis  of  Actigraph 
data.  The  Fatigue  Avoidance  Scheduling  Tool  (FAST),  developed  by  NTI,  Inc.,  was  used 
to  model  the  performance  effectiveness  of  the  participants  based  on  the  WAM  data. 
Finally,  ARES  software  used  on  the  handheld  PDA  devices,  developed  by  Dr.  Timothy 
Elsmore,  provides  researchers  with  a  performance  data  point  based  on  measurements 
taken  from  tasks  conducted  by  participants  throughout  the  study. 

I.  Actigraphs  and  ACT  Millennium  Software 

There  are  several  types  of  commercial  WAMs  that  monitor  activity  levels 
available  on  the  market.  NPS  research  teams  have  used  the  Octagonal  Sleepwatch® 
(Sleep- O  Model)  Actigraph  and  associated  Altigraph  reader  interface  made  by  PDC  (see 
Figure  3).  The  Actigraph  contains  an  accelerometer  that  can  be  set  to  record  activity  in 
epochs  varying  from  one  second  up  to  one  minute.  All  Actigraphs  used  in  NPS  studies 
were  set  to  collect  data  in  one -minute  epoch  lengths.  The  ACT  Millennium  software  is 
the  Windows  based  computer  program  used  to  initialize  the  Actigraphs  and  download  the 
data  from  them.  The  watch  is  placed  on  an  Actigraph  reader  (shown  in  Figure  3)  that 
allows  communication  between  the  ACT  Millennium  software  and  the  Actigraph.  Focal 
time,  type  of  Actigraph,  sampling  modes  (e.g..  Zero  Crossing  mode.  Time  Above 
Threshold  mode,  and  Proportional  Integrating  Measure  mode),  epoch  length,  Actigraph 
identification  number,  when  to  start  recording  data,  estimated  runtime,  and  a  header 
preview  are  all  reviewed  and  set  during  initialization.  ACT  Millennium  also  has  a 
diagnostics  function  that  allows  the  research  team  to  check  the  settings  of  the  Actigraph 
reader  and  to  ensure  the  accurate  transmission  of  data. 
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Figure  3.  Octagonal  Sleepwatch®  and  Reader  Interface 

Figure  4  shows  the  ACT  Millennium  start  up  screen.  ACT  Millennium  allows  the 
Actigraph  data  to  be  saved  with  a  .DAT  file  extension  which  can  be  read  by  ACT 
Millennium  software  or  with  the  .AMI  file  extension  that  can  be  read  by  another  AMI 
software  program  called  Action  W. 


Figure  4.  ACT  Millennium  Start  Up  Screen 
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2.  Initial  Processing  of  Actigraph  Data:  Action  W 

Ambulatory  Monitoring,  Inc.  also  markets  the  Action  W  software.  Action  W  allows 
visualization  of  the  activity  levels  for  the  entire  data  collection  period.  A  screen  capture 
of  an  individual’s  sleep  data  are  displayed  in  Figure  5.  Action  W  scores  the  sleep  and 
wake  periods  and  offers  manual  trimming  of  the  data  if  the  researcher  deems  it  necessary. 
The  red  underline  indicates  sleep  and  the  green  shaded  portion  represents  down  periods. 
Basic  statistics  can  be  run  using  Action  W  such  as  total  sleep  minutes,  wake  minutes  and 
percentages  of  time  spent  sleeping.  The  Cole-Kripke  algorithm  is  used  when  scoring  the 
sleep  and  wake  periods.  This  method,  developed  in  1988  and  revised  in  1992,  uses  a 
weighted  average  of  the  previous  four  minutes,  the  current  minute,  and  the  next  two 
minutes  in  order  to  determine  whether  the  current  minute  should  be  labeled  as  ‘sleep’  or 
‘wake.’  The  researcher  can  code  the  data  to  override  the  automatic  scoring  of  the 
accelerometer  to  allow  for  the  correction  of  the  more  obvious  discrepancies  that  may 
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Figure  5  Action  W  Linear  Actigraph  View  of  Sample  Individual’s  Data 
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result  from  using  the  weighted  average.  Action  W  allows  the  researcher  to  save  a  file 
with  the  file  extension  .AMI  for  use  again  in  Action  W  or  as  an  epoch  by  epoch  (.EBE) 
format  for  use  with  EAST. 

3.  Fatigue  Avoidance  Scheduling  Tool  (FAST) 

FAST  is  a  computer  program  developed  by  NTI,  Inc.  designed  to  allow 
schedulers  to  estimate  the  effects  of  various  schedules  on  human  performance.  It  also 
allows  WAM  data  to  be  imported  into  a  FAST  schedule.  Using  the  Actigraph  watch,  for 
example,  researchers  at  NFS  have  been  able  to  “clean”  the  collected  data  using  the 
Action  W  tool  and  then  import  that  data  to  the  FAST  tool.  This  allows  the  researcher  to 
see  the  modeled  performance  effectiveness  levels  of  the  participant  throughout  the  course 
of  the  study.  FAST  gives  estimates  of  an  individual’s  predicted  effectiveness  based  on 
the  SAFTE  model  described  earlier.  It  uses  the  SAFTE  algorithm  created  by  Dr.  Steve 
Hursh  (SAIC)  in  conjunction  with  the  Walter  Reed  Army  Institute  of  Research 
(WRAIR).  The  schematic  of  the  SAFTE  Model  is  diown  in  Figure  6.  It  integrates 
quantitative  information  about  circadian  rhythms  in  metabolic  rate,  cognitive 
performance  recovery  rates  associated  with  sleep,  and  cognitive  performance  decay  rates 
associated  with  wakefulness,  and  cognitive  performance  effects  associated  with  sleep 
inertia  (Hursh  et.  al,  2003).  The  software  also  allows  the  research  team  to  pre-condition 
the  starting  condition  for  an  individual  to  adjust  for  differences  in  sleep  prior  to  the  actual 
data  collection  period.  This  allows  the  level  of  the  individual’s  sleep  reservoir  to  be  set 
based  on  more  realistic  conditions.  FAST  then  analyzes  each  minute  of  data  imported 
and  graphically  represents  the  individual’s  sleep  efficiency. 
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Schematic  ofSAFTE  Model 
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Figure  6.  Schematic  of  the  SAFTE  Model  (From:  Hursh  et.  al,  2003) 


A  screen  capture  with  explanation  is  shown  in  Figure  7.  Sleep  intervals  and  work 
periods  are  depicted  on  the  horizontal  axis.  The  vertical  axis  represents  the  effectiveness, 
and  the  right  hand  side  shows  a  blood  alcohol  equivalent  measure.  The  main  display  is 
shown  in  colors  that  represent  degrees  of  effectiveness.  The  upper  section  is  green 
indicating  excellent  effectiveness,  followed  by  yellow  indicating  reduced  effectiveness. 
The  bottom  portion  is  pink/red  and  indicates  less  than  65%  effectiveness.  A  dotted  line  is 
shown  at  the  78%  effectiveness  level;  a  level  chosen  by  the  U.S.  Air  Force  as  the  “floor” 
for  predicted  effectiveness  for  their  pilots.  Additional  tools  allow  the  researcher  to  view 
and  manipulate  the  data  in  many  different  ways,  such  as  minute-by-minute  effectiveness 
or  by  altering  start  times  and  durations  of  wake  periods.  If  the  Action  W  data  are  not 
accurately  trimmed  to  identify  true  sleep/wake  periods,  the  FAST  data  will  be  skewed. 
Requiring  the  individual  participants  to  maintain  a  log  of  their  activity  during  the  data 
collection  period  allows  researchers  to  trim  the  collected  Actigraph  data  effieiently.  The 
quality  of  these  logs  may  determine  the  accuracy  of  the  FAST  modeled  outputs  of 
performance  effectiveness. 
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Figure  7  FAST  Display 
4.  ARES  (ANAM™  Readiness  Evaluation  System) 

ARES  is  automated  neuropsychological  testing  software  for  operation  on 
handheld  PDAs  that  use  the  Palm  Operating  System  (OS)  Software  Version  3.5  or  later 
that  was  developed  by  Dr.  Timothy  Elsmore.  The  system  is  designed  to  run  test  batteries 
that  are  intended  to  meet  specific  clinical,  operational,  or  research  goals.  These  tests 
include  Sleep  Scale,  Mood  Scale  2,  Simple  Reaction  Time,  Code  Substitution  (Eeaming), 
Math  Processing,  Matching  to  Sample,  Eogical  Relations,  Code  Sub  (Delayed  Recall), 
and  Memory  Search.  The  ARES  will  operate  on  a  wide  range  of  PDAs  including  those 
with  black  and  white  or  color  displays.  The  start-up  screen  is  shown  in  Eigure  8. 
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Figure  8.  PDA  Device  (Palm  mlOO)  with  ARES  Software 

The  ARES  allows  multiple  users  on  a  single  PDA,  and  includes  a  variety  of 
standard  test  batteries  and  a  means  to  install  custom  batteries.  The  data  saved  on  the 
PDA  can  be  downloaded  to  a  desktop  data  management  program,  DataMan,  using  a 
direct  serial  link,  USB  connection  or  wireless  modems.  The  data  can  be  downloaded 
using  the  HotSync  Manager  associated  with  most  PDA  software  or  by  importing  directly 
from  the  handheld  device  to  the  DataMan  software.  When  using  HotSync,  ARES  data 
are  stored  in  a  Palm  Database  (.PDB)  file  format  which  in  turn  can  be  accessed  using  the 
DataMan  software.  The  DataMan  program  has  limited  data  analysis  functions  and  allows 
the  data  to  be  copied  easily  into  Microsoft  Excel  or  Microsoft  Access  tables  for  further 
analysis.  The  NPS  Human  Systems  Integration  Eab  (HSIE)  currently  uses  the 
Handspring  Treo  90  as  the  PDA  device  of  choice  based  on  cost  and  functionality. 

E.  KNOWLEDGE  VALUE  ADDED  METHODOLOGY 

To  evaluate  the  reengineered  process  as  compared  with  the  current  process,  the 
Knowledge  Value  Added  (KVA)  methodology  was  used.  Developed  for  use  in  an 
economic  setting,  this  KVA  methodology  is  often  used  to  determine  the  value 
information  technology  adds  to  a  given  process.  Other  methods  of  assessing  value 
include  Cost-Based  Approaches,  Market  Approaches,  Income  Approaches,  and  Real 
Options  Approaches.  Because  of  the  nature  of  this  comparative  analysis,  it  was 

necessary  to  choose  a  methodology  that  allowed  a  valuation  for  knowledge.  Because 
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revenue  is  not  generated  in  these  processes,  a  surrogate  measurement  was  required.  KVA 
provided  the  means  to  measure  the  inputs  to  the  current  process  while  capturing  the  given 
outputs  and  compare  this  to  those  of  the  reengineered  process. 

KVA  is  based  on  the  principle  of  replication  that  states  “given  that  we  have  the 
knowledge  necessary  to  produce  the  change,  then  we  have  the  amount  of  change 
introduced  by  the  knowledge”  (Housel  and  Bell,  2001).  If  a  process  input  is  the  same  as 
its  output,  there  was  no  change,  therefore,  no  value  has  been  added.  Representing  a 
process  as  a  function  P(X)  where  P(X)  =  Y,  it  follows  that  if  X  =  Y,  there  was  no  change 
and  no  value  added  by  the  process.  Value  is  assumed  to  be  proportionate  to  change. 
Change  can  be  measured  by  the  amount  of  knowledge  required  to  make  the  change.  By 
definition,  value  is  proportionate  t)  change  which  is  proportionate  to  the  knowledge 
required  to  make  the  change.  To  compare  the  current  sleep  study  process  to  the 
reengineered  process,  it  was  necessary  to  measure  the  amount  of  knowledge  required  to 
produce  the  output  of  the  core  sub -processes  and  measure  the  cost  of  using  this 
knowledge  to  find  the  Return  on  Knowledge  (ROK)  as  the  factor  of  comparison.  ROK  is 
the  amount  of  knowledge  divided  by  the  cost.  This  ROK  is  a  corporate  performance  ratio 
that  gives  order  of  magnitude  comparisons  for  all  of  the  processes.  The  amount  of 
knowledge  is  determined  by  process  analysis  and  interviews  with  subject  matter  experts 
to  quantify  the  learning  time  (relative  and  actual)  for  each  process.  Process  instructions 
or  bits  required  to  produce  process  outputs  are  other  measurement  techniques  to  obtain 
the  amount  of  knowledge.  The  cost  is  calculated  by  multiplying  the  cost  for  the  resource 
by  the  amount  of  time  required  to  use  the  knowledge  to  produce  an  output.  The  resulting 
cost  number  is  applied  directly  to  the  output  for  the  process. 

The  seven  step  method  (Housel  and  Bell,  2001)  to  find  the  ROK  for  both  the 
current  and  reengineered  process  was  used  to  measure  how  much  the  productivity 
improved  using  information  technology  from  the  current  or  “AS -IS”  process  to  the 
reengineered  or  “TO-BE”  process.  The  first  step  is  to  identify  the  core  process  and  sub¬ 
processes.  In  step  2,  common  units  of  measure  for  learning  time  are  established. 
Calculation  of  the  learning  time  to  execute  each  of  the  sub -processes  is  step  3.  In  step  4, 
the  sampling  time  period  is  determined  to  capture  a  representative  sample  of  the  core 

process’s  final  product  or  output.  In  step  5,  the  learning  time  for  each  sub-process  is 
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multiplied  by  the  number  of  executions  of  that  sub -process  during  the  sample  period  to 
get  the  numerator  for  the  ROK.  Similarly,  in  step  6  the  denominator  for  the  ROK  is 
determined  by  calculating  the  costs  for  each  sub-process.  In  the  final  step,  the  ROK  is 
calculated  and  the  results  are  interpreted. 
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m.  CURRENT  PROCESS  EOR  DATA  COLLECTION  AND 
ANALYSIS  IN  RECENT  NPS  EATIGUE  STUDIES 


A.  OVERVIEW 

Because  the  reengineered  process  for  fatigue  studies  conducted  by  HSIL  at  NPS 
serves  as  the  Proof  of  Concept  for  the  data  collection  and  storage  methodology  for  human 
research  in  field  environments,  it  was  necessary  to  articulate  the  current  or  “AS -IS” 
process  used  in  recent  NPS  fatigue  studies.  The  Knowledge  Value  Added  (KVA) 
approach  to  this  process  reengineering  effort  provides  an  objective  method  for  measuring 
the  amount  of  value  a  given  component  process  adds  as  well  as  the  cumulative  value  of 
the  entire  process.  This  enabled  the  comparative  analysis  of  the  current  or  “AS -IS” 
process  to  the  process  after  reengineering  efforts  or  the  “TO-BE”  process.  To  compare 
this  value  added,  we  had  to  calculate  the  Return  on  Knowledge  (ROK)  at  the  process  and 
sub-process  levels  for  both  the  “AS -IS”  process  and  the  reengineered  or  “TO-BE” 
process. 

In  this  chapter,  the  seven  step  KVA  process  was  used  to  determine  the  “AS-IS” 
ROK.  Because  the  “AS-IS”  model  is  a  reflection  of  the  current  process  used  by  HSIE  at 
NPS,  summaries  of  the  techniques  used  in  two  recent  NPS  studies  are  provided  to  lay  the 
foundation  for  the  “AS-IS”  model  development.  A  review  of  the  seven  step  process  is 
described  and  then  applied  to  determine  the  ROK  for  the  “AS-IS”  process.  Einally,  an 
assessment  of  the  KVA  analysis  results  follows  with  the  identification  of  the  focus  areas 
for  the  reengineering  effort,  revised  data  collection  and  storage  methodologies. 

B.  RECENT  NPS  FATIGUE  STUDIES 

Two  recent  studies  conducted  by  NPS  thesis  students  included  the  U.S.  Navy 
Recruit  Training  Command  at  Great  Eakes,  IE  (NTCGE)  Study  (Baldus,  2002)  and  the 
USS  JOHN  C.  STENNIS  Study  (Nguyen,  2002).  While  the  objectives  of  these  studies 
differed,  the  methods  used  for  data  collection  were  similar.  In  both  studies,  Actigraph 
watches  were  used  to  monitor  participants’  activity  levels  while  other  data  were  collected 
on  the  participants.  In  the  Great  Eakes  study,  recruit  test  scores,  demographic  and  other 
data  were  collected  via  various  databases  through  local  points  of  contact  at  Great  Eakes, 
Naval  Training  Center  (Baldus,  2002).  Similarly,  in  the  USS  JOHN  C.  STENNIS  study, 
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demographic  data  and  data  from  a  momingness/eveningness  survey  (Nguyen,  2002)  were 
collected.  To  collect  his  information,  LT  Nguyen  provided  paper  surveys  for  his 
participants  to  complete.  In  addition,  the  USS  JOHN  C.  STENNIS  study  participants 
were  instructed  to  maintain  daily  paper  journals  to  record  their  sleep  behavior  as  well  as 
to  record  their  oral  temperature  readings.  In  the  Great  Lakes  study,  no  journals  or  logs 
were  maintained  by  the  participants.  Because  of  the  operational  environments  of  these 
two  studies,  the  researchers  wanted  to  minimize  the  obtrusiveness  of  their  data  collection 
and  as  a  result,  made  every  effort  to  mitigate  the  burden  on  participants  while  optimizing 
the  data  collected. 

The  data  collection  methodology  for  both  of  these  studies  required  a  tremendous 
amount  of  human  intervention  by  the  researcher.  After  the  time  consuming  effort 
involved  in  transposing  data  from  manual  pen  and  paper  entries  from  the  USS  JOHN  C. 
STENNIS  participants  and  the  printed  data  sheets  from  the  Great  Lakes  participants  to 
electronic  MS  Excel  spreadsheets,  many  hours  were  spent  cleaning  the  data.  Because  the 
data  from  the  USS  JOHN  C.  STENNIS  study  were  collected  and  recorded  before  the  start 
of  the  Great  Lakes  study,  the  lack  of  confidence  in  participants’  ability  to  maintain 
accurate  logs  under  operational  constraints  drove  LT  Baldus’  decision  to  exclude 
participants  from  recording  daily  activity.  Additional  variables  were  of  interest  to  both 
researchers  but  the  uncertainty  of  the  collection  methods  prevented  their  inclusion  in  the 
studies.  Eor  example,  both  researchers  had  desires  to  collect  performance  data  but  had  no 
feasible  means  to  do  so.  While  performance  testing  methods  were  available,  the 
unfamiliar  logistics  involved  in  using  a  viable  collection  method  was  a  major  obstacle 
preventing  the  researchers  from  collecting  performance  data. 

C.  KVA  METHODOLOGY:  THE  SEVEN  STEP  APPROACH  TO 

DETERMINING  RETURN  ON  KNOWLEDGE 

In  the  audit  of  the  “AS-IS”  process,  we  evaluated  the  Return  on  Knowfedge 
(ROK)  at  the  process  and  sub-process  levels.  To  determine  ROK,  we  measured  learning 
time  and  cost.  Learning  time  was  used  as  a  surrogate  to  measure  knowledge  embedded  in 
the  processes.  Learning  time  is  the  amount  of  time  it  would  take  for  someone  to  learn 
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how  to  do  each  activity  in  the  process.  The  cost  refers  to  how  much  money  is  required  to 
use  this  knowledge  each  time  the  process  or  sub-processes  are  executed.  After  gathering 
the  learning  time  and  cost  information,  the  Return  on  Knowledge  (ROK)  was  calculated. 

The  following  seven- step  methodology  measures  processes  in  terms  of  their 

ROK. 

STEP  1 :  Identify  the  core  process  and  its  component  processes 

STEP  2:  Establish  common  units  to  measure  learning  time 

STEP  3:  Calculate  learning  time  to  execute  each  component  process 

STEP  4:  Designate  a  time  period  long  enough  to  capture  a  representative 

sample  of  the  organization’s  final  product/service  outputs  following  common  statistical 
sampling  practices 

STEPS:  Multiply  the  learning  time  for  each  component  process  by  the 

number  of  times  the  component  executes  during  the  sample  period 

STEP  6:  Calculate  the  total  cost  to  produce  the  output  for  each  component 

STEP  7 :  Compute  the  ROK  for  each  component  process 

Since  this  method  relies  on  the  estimations  of  Subject  Matter  Experts  (SMEs)  for 
process  learning  time  to  gain  a  high-level  ROK  assessment  of  a  given  process,  it  is  often 
useful  to  use  another  surrogate  for  knowledge  such  as  process  instructions  to  ensure 
validity  of  the  analysis.  In  this  Proof  of  Concept  effort,  we  consulted  with  the  SMEs  and 
developed  a  process  model  using  actual  learning  times  for  each  sub-process.  The 
granularity  of  this  effort  provided  sufficient  evidence  to  the  stakeholders  at  NPS  that 
further  assessments  were  not  required.  The  same  level  of  granularity  was  applied  to  the 
reengineered  “TO-BE”  process  allowing  viable  comparisons  to  be  made. 

D.  CURRENT  PROCESS  MODEL 

To  clearly  represent  the  process  model  for  the  past  NPS  fatigue  research  studies, 

we  interviewed  Dr.  Nita  Miller,  head  of  the  Human  Systems  Integration  Eab  (HSIE)  at 

NPS,  and  several  of  her  thesis  students  who  have  conducted  data  collection  and  analysis 

for  fatigue  studies.  Additional  interviews  were  conducted  with  other  DoD  SMEs  who  are 

involved  in  similar  processes  of  collection  and  analysis.  Observations  of  the  NPS 
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researchers  were  also  recorded  during  the  latest  studies  to  allow  the  creation  of  a 
representative  model  of  the  “AS-IS”  process.  Applying  STEP  1  of  the  seven  step  KVA 
methodology,  the  core  process  of  Conducting  a  Fatigue  Research  Study  was  identified. 
Three  basic  phases  were  determined  as  sub -processes  of  the  model:  1)  Collect  Data,  2) 
Prepare  Data  for  Analysis,  and  3)  Analyze  and  Publish  Data. 

1.  Collect  Data  Sub- Process 

In  the  Collect  Data  Sub-Process  shown  in  Figure  9  below,  the  activities  are 
depicted  with  circles  while  the  outputs  of  those  activities  (or  processes)  are  indicated  by 
squares.  The  sub-process  begins  with  a  requirement  for  research  resulting  in  a  request  for 
the  HSIF  to  conduct  a  study.  Included  in  this  input  request  are  the  study  parameters: 
purpose  of  the  study,  permission  for  study  from  customer  requesting  research, 
identification  of  the  research  team,  and  methods  of  data  collection.  Because  the  model 
for  NPS  research  centers  on  students  collecting  and  analyzing  the  data,  training  of  the 
research  team  is  essential  for  each  study  conducted.  This  Train  Research  Team  Activity 
focuses  on  teaching  research  methods  to  the  research  team  and  includes  instruction  on 
study  design,  methods  of  data  collection  and  analysis,  and  other  unique  considerations  of 
conducting  human  research  in  operational  environments.  The  output  of  the  activity  is  a 
Trained  Research  Team. 


Completed 
Paper 
Surveys, 
Activity/ 
Temp/Sleep 
Rating  Log 

Watches  with 
stored  collected 
data 

(thermometers) 

Figure  9.  Collect  Data  Sub- Process 


The  next  phase  of  this  sub-process  is  to  define  the  conditions  under  which  the  data 
will  be  collected.  Under  the  Preparing  for  Data  Collection  Activity,  there  are  several 
tasks  required  including: 
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•  Design  the  Study 

•  Identify  Funding  Requirements 

•  Create  Demographic  Survey 

•  Type  Lark  and  Owl  Survey 

•  Create  Activity /Temp/Sleep  Rating  Log 

•  Type  Consent  Forms 

•  Produce  Institutional  Review  Board  Proposal 

The  output  documents  from  these  tasks  are  the  completed  study  design,  approved 
Institutional  Review  Board  Proposal,  funding  requirements,  typed  surveys,  logs,  and 
consent  forms. 

Just  before  data  collection  actually  starts,  the  required  equipment  which  is  to  be 
used  in  the  collection  is  prepared.  The  tasks  for  this  Prepare  Equipment  Activity  include 
initializing  the  Actigraphs,  obtaining  thermometers  when  participants’  temperatures  will 
be  recorded,  and  making  copies  of  all  surveys  and  forms  for  all  participants.  The  output 
is  all  of  the  paperwork  and  equipment  prepared  for  collection.  Once  on  location,  the 
research  team  then  has  to  identify  the  study  participants.  Because  of  operational 
constraints,  pure  random  sampling  is  almost  never  an  option  but  the  researchers  try  to 
select  a  sample  of  participants  that  will  be  representative  of  the  entire  group.  Once 
selected,  the  participants  fill  out  consent  forms  enabling  the  research  team  to  conduct  the 
study. 

The  next  activity.  Distribute  and  Collect  Surveys,  Equipment,  and  Eogs,  includes 
both  issuing  the  equipment  and  the  instructions  required  of  the  participants  as  well  as  the 
collection  of  the  equipment  upon  completion  of  the  data  collection  phase.  The  output  of 
these  tasks  includes  the  surveys  and  logs  completed  by  participants  as  well  as  the 
Actigraphs  with  recorded  actigraphy  in  their  memory,  and  the  thermometers.  Once  all 
equipment  and  data  are  collected,  the  Collect  Data  Sub-Process  is  complete. 

2.  Prepare  Data  for  Analysis  Sub- Process 

With  the  collected  equipment,  paper  surveys,  and  logs,  the  research  team  returns 
to  HSIE  at  NPS  and  begins  the  Prepare  Data  for  Analysis  Sub -Process  represented  in 
Eigure  10.  The  inputs  to  this  sub-process  come  from  the  Data  Collection  Sub-Process 

outputs  e.g.,  Actigraphs  with  recorded  data  and  completed  surveys  and  logs. 
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Figure  10.  Prepare  Data  for  Analysis 


The  first  activities  of  this  sub-process  include  transferring  the  data  from  the 
Actigraphs  and  paper  recordings  to  an  electronic  format  that  allows  the  researcher  to  then 
“clean”  the  raw  data.  The  data  from  the  paper  surveys  and  logs  must  be  manually  input 
to  a  Microsoft  (MS)  Excel  spreadsheet.  The  data  from  the  sleep  watches  has  to  be 
downloaded  to  a  computer  using  the  Actigraph,  the  Actigraph  Interface  Unit,  and  Act 
Millennium  software.  Data  from  each  Actigraph  are  saved  with  the  file  extensions  .DAT 
(Act  Millennium  Data  format)  and  .AMI  (AMI  without  Application  format)  for  use  with 
the  Action  W  software.  Once  these  raw  data  files  have  been  established,  the  next 
activities  include  “cleaning”  the  data. 

The  data  from  surveys  and  logs  have  to  be  checked  manually  for  data  entry  errors. 
Additionally,  the  quality  of  the  logs  maintained  by  the  participants  is  determined  by  the 
researchers.  The  participants  often  forget  to  record  their  data  or  become  so  overwhelmed 
with  operational  events  that  their  logs  are  often  incomplete  or  inaccurately  filled  out.  If 
there  are  too  many  errors  and  they  cannot  be  rectified,  researchers  may  have  to  remove  an 
entire  participant’s  data  set  from  the  study.  Additionally,  the  tedious  nature  of  manually 
entering  data  from  paper  journals  to  electronic  media  often  requires  the  research  team  to 
review  the  data  entries  several  times  to  ensure  accuracy. 

The  Actigraph  raw  data  are  imported  to  the  Action  W  software.  Based  on  the 
algorithm  for  sleep  scoring  activity  level.  Action  W  automatically  codes  the  Actigraph’ s 
recorded  activity  epochs  as  sleep  or  wake  episodes.  An  example  of  raw  data  of  a 
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participant  that  has  been  automatically  coded  by  Action  W  software  is  shown  in  Figure 
1 1 .  The  coded  data  are  compared  with  the  sleep/wake  log  of  the  participant.  If  there  are 
discrepaneies,  the  researcher  has  to  determine  the  best  code  for  the  data.  This 
subjectivity  is  necessary  to  overcome  limitations  of  the  equipment. 


Figure  11.  Automatically  Scored  WAM  Data  Using  Action  W 


In  this  example,  the  participant  logged  that  he  had  taken  off  his  WAM  for  a 
shower  during  the  times  circled  in  Figure  12.  Because  the  activity  levels  are  at  or  near 
zero,  the  Action  W  scores  these  periods  as  sleep  epochs  (indicated  by  the  red  line  beneath 
the  recorded  time  periods). 


Figure  12.  Circled  Times  the  Participant  Removed  His  Actigraph  for  Showers 
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The  researcher  can  manually  recode  these  epochs  as  sleep,  as  shown  in  Figure  13, 
to  allow  more  a  more  accurate  depiction  of  the  participant’s  behavior.  Once  “cleaned,” 
the  file  is  saved  with  file  extensions  .AMI  (for  reuse  with  Action  W)  and  as  .EBE  (epoch 
by  epoch  for  use  with  the  FAST  software).  Any  data  that  had  been  corrupted  due  to 
mechanical  or  human  errors  are  also  identified  and  may  be  removed  from  the  study 
altogether.  Action  W  also  calculates  sleep  statistics  for  each  data  file.  The  output 
products  of  this  activity  are  the  data  files  themselves,  the  statistics  reports  for  each 
participant  and  the  MS  Excel  spreadsheets  with  recorded  survey  and  log  data. 
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Figure  13.  Manually  Recoded  Shower  Times  as  Wake  Epochs 


The  Actigraph  data  are  then  processed  in  two  ways.  First,  the  researcher  exports 
the  sleep  statistics  from  the  Action  W  data  files  into  a  MS  Excel  spreadsheet  and 
manually  manipulates  these  into  a  format  compatible  with  data  analysis  tools  for  use  at  a 
later  stage  in  the  process.  The  second  way  the  data  are  processed  is  using  the  EAST 
modeling  tool.  The  researcher  sets  up  a  schedule  encompassing  the  period  of  the  study 
and  pre-conditions  the  three  previous  days  of  the  data  collection  period  to  match  a 
realistic  sleep/wake  cycle  of  the  participant.  If  enough  data  were  collected,  the  researcher 
can  use  the  first  three  days  of  collected  data  to  automatically  set  the  pre-conditions  for  the 
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participant.  Once  a  FAST  schedule  is  defined,  the  researcher  imports  the  .EBE  Actigraph 
file  of  each  participant  to  this  schedule  and  is  able  to  model  the  performance 
effectiveness  during  the  period  of  the  study.  The  researcher  compares  the  EAST  analysis 
to  the  logs  and  surveys  to  identify  correlations.  Any  required  data  from  the  EAST  models 
are  recorded  in  MS  Excel  spreadsheets  to  allow  for  further  data  analysis  by  the 
researcher.  The  outputs  of  this  sub -process  include  all  Action  W  sleep/wake  periods 
screen  captures,  EAST  performance  effectiveness  model  screen  captures,  as  well  as  the 
“cleaned”  MS  Excel  spreadsheets  prepared  for  export  to  data  analysis  tools. 

3.  Analyze  and  Publish  Sub-Process 

The  data  from  the  previous  sub -process  are  analyzed  and  the  comprehensive  analysis  is 
published  in  the  Analyze  and  Publish  Sub-Process  shown  in  Eigure  14.  The  “cleaned” 
data  are  analyzed  using  MS  Excel  tools  as  well  as  exported  to  SPSS  (statistical  software 
package  used  by  HSIE  researchers)  for  further  analysis.  Once  in  SPSS,  the  data  have  to 
be  properly  formatted  for  analysis.  The  output  of  the  analysis  includes  the  results  of 
various  statistical  tests  performed  on  the  data  in  the  form  of  tables  and  graphs.  The 
research  team  then  combines  all  of  their  efforts  and  writes  their  thesis  using  the  outputs 
of  these  analysis  tools  and  the  outputs  from  other  sub -processes.  The  final  output  of  the 
whole  fatigue  study  process  is  the  published  thesis  where  conclusions  are  drawn  based  on 
the  analysis  conducted. 


Figure  14.  Analyze  and  Publish  Sub-Process 


E.  RETURN  ON  KNOWLEDGE  FOR  “AS -IS”  PROCESS  USING  KVA 
ANALYSIS 

In  order  to  compare  the  current  process  with  any  newly  engineered  process,  a 
framework  of  comparison  has  to  be  established.  Using  the  KVA  methodology,  we  are 
able  to  calculate  a  ROK  (Return  on  Knowledge)  of  the  current  sub -processes  in  terms  that 
we  can  compare  with  the  reengineered  process.  Additionally,  KVA  allows  us  to  identify 

the  shortcomings  of  the  current  process.  The  numerator  in  the  ROK  ratio  is  the  total 
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amount  of  learning  time  required  to  execute  a  process.  The  denominator  is  the  cost  of 
executing  the  process  (or  firing  that  knowledge).  A  summary  of  the  core  process  and 
sub-processes  identified  in  STEP  1  are  shown  in  Figure  15.  Following  STEP  2,  the 
common  units  used  to  calculate  for  learning  time  in  hours  were  established.  The  number 
of  hours  required  to  learn  to  execute  each  task  was  calculated  as  part  of  STEP  3.  Where 
applicable,  the  learning  time  of  associated  IT  products  when  used  in  the  execution  of  the 
task  was  also  added  to  the  total  learning  time.  Following  STEP  4,  the  representative 
sampling  time  period  was  determined  to  be  one  year.  To  determine  the  weighted  learning 
time  for  each  sub-process  according  to  STEP  5,  the  learning  time  required  for  one 
execution  of  each  task  was  determined  through  interviews  with  SMEs. 


Figure  15.  “AS-IS”  Process 


Some  tasks  were  executed  more  than  one  time.  For  example,  when  preparing 
equipment  for  the  study,  the  task  of  initializing  the  Actigraph  watch  had  to  be  repeated 
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for  every  participant  on  the  study.  If  the  task  was  executed  more  than  one  time,  then  the 
knowledge  for  that  task  fired  more  than  one  time.  The  number  of  times  the  knowledge 
was  fired  in  the  given  sample  period  is  referred  to  as  the  weighted  learning  time.  In  the 
NPS  studies,  the  average  number  of  studies  completed  in  each  sample  period  of  one  year 
was  one  and  the  average  number  of  participants  identified  for  a  study  was  25.  However, 
due  to  human  and  equipment  errors,  the  SMEs  estimated  that  on  average  20%  of  data 
from  the  participants  were  not  usable  in  the  study.  In  other  words,  the  average  number  of 
participants  with  adequate  data  was  estimated  to  be  20  per  study.  The  weighted  learning 
time  is  only  calculated  for  the  amount  of  successful  executions  of  the  tasks.  Therefore,  in 
a  task  such  as  initializing  the  Actigraphs,  the  learning  time  for  one  execution  of  the  task 
was  multiplied  by  20  (the  number  of  successful  executions)  to  get  the  weighted  learning 
time  for  the  task.  After  all  of  the  weighted  learning  times  for  each  task  were  calculated, 
they  were  added  together  for  a  cumulative  learning  time  in  hours  for  the  activity. 
Correspondingly,  the  total  learning  times  for  each  of  the  activities  were  added  together 
representing  the  total  learning  time  for  the  sub-process.  In  STEP  6,  the  cost  to  execute 
each  task  was  calculated.  The  hourly  wage  of  the  researcher  was  conservatively 
estimated  at  $22  (based  on  the  annual  base  pay  of  an  0-3  in  the  active  service).  The 
cycle  time  or  the  amount  of  time  for  the  researcher  to  execute  the  task  was  calculated. 
Because  the  cost  included  the  total  number  of  times  the  task  was  executed,  the  cost  for 
the  task  included  the  times  the  task  was  executed  but  the  data  was  not  used.  Eor  example, 
if  the  researcher  had  to  initialize  25  Actigraphs,  the  cost  calculated  for  the  task  was  the 
cycle  time  to  initialize  an  Actigraph  multiplied  by  25  watches.  After  the  cost  for  each 
task  was  determined,  the  total  cost  for  each  activity  was  calculated  by  adding  the  cost  for 
each  task.  Similarly,  the  total  cost  for  each  sub -process  was  then  calculated  by  adding 
the  total  costs  of  each  activity  included  in  this  sub-process.  To  determine  the  ROK 
according  to  STEP  7,  the  ratio  of  weighted  learning  time  divided  by  cost  was  calculated. 
A  summary  of  the  calculated  weighted  learning  times  and  total  costs  to  produce  the  ROK 
for  each  sub-process  is  shown  in  Table  1.  (Eor  a  comprehensive  table  of  calculations  by 
task,  see  Appendix  A.)  At  NPS,  these  research  studies  are  conducted  by  students 
working  under  a  principle  investigator/professor.  As  a  result,  the  ability  to  maintain 
corporate  knowledge  is  extremely  difficult.  Once  a  student  becomes  familiar  with  how  to 
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complete  the  tasks,  he  graduates  resulting  in  a  loss  of  knowledge  he  acquired  while 
executing  the  tasks  for  the  study.  Because  of  this  major  factor,  the  ROK  of  the  whole 
process  is  inherently  low.  The  reengineering  effort  is  critical  to  try  to  streamline  the 
process  to  make  it  as  efficient  as  possible  but  also  to  try  to  capture  some  of  the 
knowledge  in  IT  tools  that  allow  NFS  to  maintain  some  of  the  corporate  knowledge  they 
are  continually  losing.  It  is  imperative  to  find  a  reengineered  process  that  will  cost  less 
and  provide  greater  return. 


Sub  Processes 

Weighted 

Learning 

Time 

(Hrs) 

Total 
Cost  ($) 

As -Is 
ROK 

Collect  Data 

289 

3300 

8.76% 

Prepare  for  Analysis 

1158 

8277.5 

13.99% 

Analyze  and  Publish 

1161 

10192 

11.39% 

TOTAL 

2608 

21769.5 

11.98% 

Table  1.  “AS-IS”  ROK 


F.  AREAS  TO  FOCUS  REENGINEERING  EFFORT 

Looking  at  the  ROK  for  each  sub-process  at  the  activity  and  task  level,  some  tasks 
were  clearly  identified  as  providing  low  return  on  knowledge.  After  consultation  with  the 
SMEs,  the  data  collection  and  data  storage  areas  of  the  process  caused  the  greatest 
concern.  The  data  collection  methods  of  the  “AS-IS”  process  are  tedious  and  error  prone. 
The  paper  logs  and  surveys  are  a  hindrance  to  participants  based  on  exit  surveys  from  the 
recent  fatigue  studies.  The  paper  entries  are  a  source  of  a  tremendous  amount  of 
additional  work.  Requiring  the  participants  to  enter  the  data  once  on  paper  and  then  the 
researcher  to  transfer  the  same  data  to  a  spreadsheet  is  an  example  of  the  inefficiency 
inherent  in  this  process.  The  extra  time  required  to  diligently  “clean”  the  data  by 
comparing  the  paper  entries  to  the  computer  entries  demonstrates  little  Return  on 
Knowledge.  The  conservative  estimate  of  a  20%  loss  of  collected  data  due  to  the 
combination  of  human  and  mechanical  errors  appears  unacceptably  high  and  avoidable. 
Additionally,  the  lack  of  data  storage  capability  is  a  significant  drawback  of  the  current 
process.  While  Dr.  Miller  is  involved  as  an  advisor  in  all  of  the  studies  conducted  by 
HSIL,  each  researcher  determines  how  he  saves  his  data  for  data  analysis.  Because  of  the 
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disparate  ways  of  storage,  the  effective  result  is  that  the  published  thesis  is  the  only 
source  of  storage  of  collected  data.  To  use  this  for  future  studies  or  conduct  any  meta¬ 
analysis  is  virtually  impossible  given  the  cost  required  to  do  so.  Considering  the 
particular  areas  of  low  ROK  in  the  “AS-IS”  model,  Figure  16  highlights  the  particular 
activities  targeted  in  the  reengineering  effort. 

After  analyzing  the  ROK  for  each  task  and  discussing  the  complexities  of  each 
with  the  subject  matter  experts,  the  focus  of  the  reengineering  effort  became  clear: 
streamlining  the  data  collection  process  and  storing  data  for  future  use. 
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Figure  16.  Areas  of  Focus  for  Reengineering  Effort 
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IV.  REENGINEERING  THE  PROCESS 


A.  OVERVIEW 

The  reengineering  effort  of  this  thesis  centered  around  creating  a  data  repository 
and  streamlining  the  data  collection  methods  using  current  information  technology.  The 
research  teams  that  conduct  the  human  performance  and  fatigue  research  at  NFS  are 
typically  thesis  students  working  towards  a  Master’s  Degree  in  Operations  Research. 
While  their  knowledge  of  data  analysis  software  is  extensive,  their  experience  with  web¬ 
page  design,  web  servers  and  databases  is  often  limited.  Therefore,  the  technology 
inserted  into  the  “TO-BE”  process  had  to  be  relatively  easy  for  the  researchers  to  learn 
while  still  maintaining  the  desired  effect  of  streamlining  the  process.  Also,  the  software 
and  hardware  we  selected  for  any  IT  implementation  had  to  be  easily  supportable.  We 
did  not  want  the  Human  Systems  Integration  Lab  to  become  dependent  on  a  specific 
vendor  where  costs  to  maintain  the  software  became  unreasonably  high  or  proprietary 
issues  would  cause  expensive  upgrades.  Every  attempt  was  made  to  use  IT  assets  that 
were  either  non-proprietary  or  that  was  supported  by  the  IT  staff  at  NFS.  This  chapter 
includes  a  consolidation  of  the  Flanning,  Requirement  Analysis,  Design  and 
Implementation  Fhases  of  the  Study  Repository  Template  database  system  development 
as  well  as  the  data  collection  technique  development.  A  summary  of  the  database  design 
created  in  Microsoft  Access  2002  and  the  collection  methods  including  the  automated 
daily  log  created  using  HanDBase  3.0  and  the  web-based  survey  forms  created  using 
Microsoft  ErontFage  2002  are  included  in  the  Implementation  Fhase. 

B.  PLANNING  (SURVEY)  PHASE 

After  extensive  study  of  the  “AS-IS”  process,  we  targeted  our  reengineering 
efforts  to  achieve  two  main  goals.  First,  we  wanted  to  allow  participants  and  researchers 
to  be  able  to  easily  record  data  with  simple  and  standard  data  collection  methods. 
Secondly,  we  wanted  to  provide  the  HSIL  with  a  data  storage  capability  that  brought 
standardization  to  the  fatigue  studies.  Along  with  both  of  these  goals,  we  also  wanted  to 
afford  researchers  the  ability  to  easily  adapt  the  collection  and  storage  means  to  suit  the 
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unique  requirement  demands  for  each  study.  In  the  planning  phase,  we  refined  these 
goals  to  prepare  for  the  design  and  implementation  of  IT  solutions  to  enhance  the  “TO- 
BE”  process. 

1.  Problem  and  Scope 

Our  challenge  was  to  develop  a  database  and  data  collection  techniques  to 
facilitate  efficient  and  timely  data  analysis  for  operational  fatigue  studies  in  an  automated 
setting  which  can  be  implemented  by  future  fatigue  research  teams  from  the  HSIL  at 
NPS.  Addressing  these  issues,  we  used  a  three  tiered  architecture  approach  to  data 
collection  and  storage  in  the  reengineering  process:  a  user  interface  layer,  a  database 
layer  and  a  web- server  layer.  The  graphical  user  interface  layer  allows  data  to  be  entered 
into  and  extracted  from  the  database  layer.  Because  both  study  participants  and 
researchers  required  access  to  the  databases,  it  was  important  to  keep  the  graphical  user 
interfaces  simple  and  self  explanatory.  By  automating  the  data  entry,  the  research  teams 
can  eliminate  the  task  of  transposing  data  from  paper  surveys  to  electronic  spreadsheets 
thus  reducing  the  number  of  errors.  The  database  layer  provides  flexibility  and  efficiency 
for  data  storage.  Additionally,  a  template  database  provides  the  HSIL  researchers  with  a 
method  of  standardization  to  maintain  data  from  all  fatigue  studies  that  currently  does  not 
exist.  The  web- server  enables  participants  to  have  the  ability  to  enter  data  using  the  web 
while  giving  researchers  the  ability  to  access  data  from  many  studies  from  a  single 
storage  area.  Templates  of  survey  designs  from  data  collection  techniques  as  well  as  the 
database  template  stored  on  the  web- server  provide  institutional  knowledge  from  which 
each  research  team  can  build  and  subsequently  add.  The  following  description 
summarizes  the  automated  functions  required  of  the  system.  The  system: 

•  Enables  a  research  team  to  record  the  parameters  of  a  study  including 
information  such  as  duration,  location  and  a  summary  of  the  study; 

•  Allows  participants  to  enter  survey  data  via  the  web  prior  to  the  start  of  the 
study  to  allow  the  research  team  to  prepare  necessary  equipment  and 
paperwork  for  the  study  ahead  of  time  as  well  as  conduct  preliminary 
analysis  of  participants  prior  to  the  study; 

•  Eacilitates  the  preparation  of  the  equipment  by  enabling  research  teams  the 
ability  to  maintain  calibration  and  initialization  status  of  the  equipment  as 
it  is  prepared  for  use  in  a  field  study; 
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•  Provides  the  capability  for  the  researcher  to  assign  equipment  to 
participants  by  serial  number  and  track  status  of  collected  equipment  and 
data  after  the  data  collection  portion  of  the  study  is  finished; 

•  Enables  participants  to  log  daily  events  easily  without  hindering  them  in 
their  operational  settings; 

•  Provides  a  means  to  collect  performance  testing  data  from  participants  in 
an  unobtrusive  manner; 

•  Provides  the  flexibility  to  easily  change  or  expand  data  collection 
requirements  to  accommodate  researchers  when  different  or  additional 
data  variables  must  be  collected;  and 

•  Supplies  the  HSIL  research  teams  at  NPS  with  standardized  templates  for 
data  collection  and  storage  methods,  expanding  and  retaining  their 
corporate  knowledge. 

2.  Constraints 

Though  each  study  conducted  by  HSIL  has  unique  requirements,  the  prototype 
system  is  designed  as  a  template  which  a  research  team  can  easily  tailor  to  suit  their 
particular  study’s  requirements.  The  participant  surveys  for  this  system  include  a 
Demographic  Survey  template  providing  information  about  the  individual  participant  and 
a  Lark  and  Owl  Survey  based  on  the  momingness/eveningness  scale  included  in 
APPENDIX  B.  The  template  also  includes  the  capability  to  collect  performance  data  on 
participants  using  the  ARES  software  designed  for  use  on  the  PDA.  Einally,  an  event  log 
template  was  created  from  a  software  database  development  package. 

C.  REQUIREMENTS  ANALYSIS  PHASE 

1.  Methods  of  Data  Collection 

Survey  input  forms  were  required  to  collect  survey  data  from  participants  via  the 
web.  Many  web-page  design  software  packages  were  evaluated  including  non¬ 
proprietary  software.  Because  the  research  teams  from  the  HSIL  generally  have  little  to 
no  web  design  skills,  it  was  important  to  choose  software  that  was  relatively  simple  to  use 
and  supported  by  NPS  IT  staff.  Microsoft  IrontPage  is  easy  to  use  and  classes  are 
routinely  offered  by  NPS  IT  staff  for  Microsoft  ErontPage.  Also,  the  software  is 
available  through  the  NPS  academic  use  license  so  no  additional  costs  for  the  HSIL  are 
incurred.  The  template  design  includes  a  generic  Demographic  Survey  and  a 
standardized  Lark  and  Owl  Survey. 
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To  maintain  equipment  statuses  and  to  simplify  the  assignment  of  equipment  to 
participants,  data  input  forms  for  the  researchers  to  use  were  also  required.  In  addition, 
the  SMEs  articulated  a  tracking  requirement  for  maintaining  a  record  of  when  assigned 
equipment  from  a  participant  was  collected  and  when  data  from  a  participant  had  been 
successfully  downloaded  to  the  database.  A  form  allowing  researchers  to  track  their 
work  progress  was  needed  to  fulfill  this  requirement.  Because  of  the  pressures  on 
researchers  conducting  a  field  study,  an  effort  was  made  to  standardize  the  data  collection 
processes  to  alleviate  some  of  their  administrative  burdens. 

In  order  to  efficiently  “clean”  participants’  Actigraph  data  and  analyze 
participants’  behavior  with  respect  to  their  work  schedules,  it  is  imperative  for 
researchers  to  be  able  to  view  a  daily  event  log  from  participants.  The  information 
collected  on  this  log  includes  events  such  as  when  a  participant  removed  his  Actigraph, 
when  he  went  down  for  sleep,  when  he  awoke  and  when  he  started  and  ended  a  work 
shift.  Because  these  events  are  logged  throughout  the  day,  using  a  PDA  to  record  these 
events  provides  an  unobtrusive  means  for  data  collection. 

ARES  performance  testing  software  for  PDAs  was  used.  The  data  collected  from 
each  test  a  participant  takes  are  stored  in  a  file  on  the  PDA  and  can  be  downloaded 
directly  into  the  accompanying  Data  Manager  software  for  ARES  data  or  stored  on  the 
PC  using  the  HotSync  function  and  later  imported  to  the  Data  Manager  software.  Once 
opened  in  Data  Manager,  the  data  can  be  copied  to  a  MS  Excel  spreadsheet  for  analysis. 

2.  Conceptual  Data  Model  for  the  Database 

After  the  thorough  analysis  of  the  current  process,  we  gained  a  comprehensive 
understanding  of  the  data  required  for  collection  and  the  processes  involved  in  using 
particular  data  for  analysis.  Through  SME  interviews,  we  also  gleaned  information 

regarding  the  potential  future  uses  for  the  data  to  ensure  we  incorporated  into  the  storage 
facility  all  pertinent  information  for  a  particular  study.  The  subsequent  paragraphs 
describe  the  requirements  for  the  database  design. 
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a.  Entity  Description 

After  clarifying  the  data  requirements,  the  following  entities  were 
modeled:  STUDY,  RESEARCH  TEAM,  PARTICIPANT,  ACTIGRAPH,  PDA,  EOG, 
ARES  DATA,  EARK  AND  OWE,  ACTGRAPH  STATUS,  PDA  STATUS,  and 
COEEECTED  EQUIPMENT  AND  DATA.  The  attributes  for  each  of  these  were  clearly 
defined.  The  ARES  DATA  attributes  include  those  defined  by  the  ARES  performance 
testing  software  designed  for  the  PDA.  The  attributes  for  the  EARK  AND  OWE  are 
based  on  the  standardized  morningness/eveningness  scale  in  Appendix  B.  The  following 
lists  these  attributes  for  each  entity: 
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PARTICIPANT 

Study  ID 

ParticipantID 

LastName 

FirstName 

Middlelnitial 

Rank 

Prior 

Specialty 

EmailAddress 

PhoneNumber 

DSN 

PagerNumber 

OfficeAddress 

City 

State 

Age 

Gender 

STUDY 

Study  ID 
Study_Name 
Study_Description 
S  tudy_B  egin_Date 
S  tudy_End_D  ate 
Study  _Duration 
Study  _Location 
Number_Participants 


ARES  DATA* 

Record 

Subject  *NOTE:  Tbe  list  of  attributes  for  ARES 

Test  is  truncated.  In  tbe  completed 

Session  database,  all  attributes  as  they  appear 

PIndex  in  tbe  DataMan  software  package 

Rep  are  included. 

TestVer 

Battery 

BatVer 

AresVer 

Seed 

Start 

Dur 

Stim 

Cor 

Err 

Imp 

Lap 

ITI 

PDA  STATUS 
PDA  ID 
ARES  Loaded 
HanDBase  Loaded 
Log  Database  Loaded 


ACTIGRAPH 

ActigrapbID 

ParticipantID 

Lastname 

PDA 

PDAID 

ParticipantID 

Lastname 

LOG 

RecordID 

ParticipantID 

Date 

Time 

Event 

Note 


RESEARCH  TEAM 

ResearcberlD 

LastName 

FirstName 

Middlelnitial 

Title_Rank 

EmailAddress 

PboneNumber 

PagerNumber 

OfficeAddress 

City 

State 

LARK  AND  OWL 

Lark  and  Owl  ID 
Participant  ID 
LastName 

Up_later_tban_planned 

B  ed_T  ime_N  o_Commitments 

Early_Exercise 

Evening_Tiredness 

Cboose_Own_Working_Hours 

Self_Consideration 

Total 

Lark  or  Owl 

ACTIGRAPH  STATUS 

Actigrapb  ID 
Battery  Last  Changed 
Times  tamp 


COLLECTED  EQUIPMENT  AND  DATA 

Lastname 
ParticipantID 
Consent  Forms  Signed 
Collected  Actigraph 
Collected  PDA 
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b.  Database  Perspectives 

The  entities  can  be  described  using  a  main  success  scenario  as  follows: 

•  A  request  to  the  HSIL  at  NFS  for  a  fatigue  study  defines  the  study 
parameters.  The  RESEARCH  TEAM  is  formed  and  contact  information 
is  collected  on  them.  The  researchers  then  record  the  STUDY  parameters 
and  coordinate  with  the  sponsoring  organization  to  identify 
PARTICIPANTS. 

•  PARTICIPANTS  complete  the  Demographic  Survey  and  the  EARK  AND 
OWE  survey  via  web  input  forms. 

•  The  RESEARCH  TEAM  updates  the  ACTIGRAPH  STATUS  and  PDA 
STATUS  and  assigns  ACTIGRAPHs  and  PDAs  to  PARTICIPANTS. 

•  The  RESEARCH  TEAM  prepares  necessary  consent  forms  and  bundles 
equipment  for  issue.  Once  at  the  location  of  the  STUDY,  the  RESEARCH 
TEAM  issues  the  equipment  (ACTIGRAPHs  and  PDAs). 

•  PARTICIPANTS  wear  the  ACTIGRAPH  watches  and  use  the  PDA  to 
EOG  daily  events  and  to  take  ARES  performance  tests  according  to  the 
prescribed  schedule. 

•  The  RESEARCH  TEAM  collects  the  equipment  and  downloads  the 
ACTIGRAPH  data  using  the  Act  Millennium  software.  They  also 
download  the  ARES  DATA  and  the  EOG  data  using  the  HotSync  function 
on  the  PDAs  and  then  copy  this  data  to  the  ARES  DATA  and  EOG  tables. 
The  COEEECTED  EQUIPMENT  AND  DATA  are  updated  to  ensure  all 
equipment  is  accounted  for  and  all  available  data  have  been  collected  from 
the  PDAs  as  well  as  the  ACTIGRAPHs  for  each  PARTICIPANT. 

•  A  EOG  report  is  generated  to  simplify  the  coding  of  actigraphy  and  the 
analysis  of  PARTICIPANT  behavior  and  performance. 

c.  Design  Methodology 

Through  analysis  of  the  current  process,  we  were  able  to  determine  which 
data  researchers  sought  to  gain  from  the  data  collection  phase  in  order  to  complete  their 
data  analysis.  Our  goal  was  to  reverse-engineer  the  database  system  from  a  bottom- up 
perspective.  We  collected  potential  attributes  including  those  already  defined  by  surveys, 
the  PDA  log  and  performance  testing  software,  linked  them  to  entities,  and  then  worked 
to  resolve  the  relationships  between  them. 

3.  Process  Data  Model 

Using  web  forms  generated  with  Microsoft  ErontPage,  the  PDA  Event  Eog,  PDA 
ARES  testing  software  as  well  as  forms  and  reports  within  the  DBMS  system,  the 
following  are  the  system  components. 
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a.  Data  Input 

•  Create/modify /delete  a  RESEARCH  TEAM. 

•  Create/modify/delete  a  STUDY. 

•  Create/modify /delete  a  PARTICIPANT. 

•  Create/modify /delete  EOG  events. 

•  Create/modify /delete  ARES  data. 

•  Track  ACTIGRAPH  STATUS. 

•  Track  PDA  STATUS. 

•  Track  COEEECTED  EQUIPMENT  AND  DATA. 

b.  Data  Maintenance 

•  Update  ACTIGRAPH  STATUS. 

•  Update  PDA  STATUS. 

•  ASSIGN  PDAs. 

•  ASSIGN  ACTIGRAPHs. 

•  Update  COEEECTED  EQUIPMENT  AND  DATA. 

c.  Reports 

•  Prepare  Participant  Information  Report  to  assist  Research  Team  with 
preliminary  data  analysis. 

•  Prepare  Assigned  Actigraph  and  Assigned  PDA  Reports  to  assist  Research 
Team  in  tracking  equipment. 

•  Prepare  Actigraph  Sign- Out  Report  to  assist  Research  Team  with 
maintaining  property  accountability. 

•  Prepare  formatted  Eog  Report  to  facilitate  “cleaning”  Actigraph  data  and 
assist  Research  Team  with  data  analysis. 

D.  DESIGN  PHASE 

1.  Designing  Data  Collection  Methods 

a.  Web  Survey  Designs 

Securing  space  on  the  HSIE  web- server,  we  created  and  saved  survey 
templates  allowing  researchers  the  ability  to  make  modifications  to  suit  each  study’s 
requirements.  We  designed  a  generic  Demographic  Survey  Eorm  and  a  Eark  and  Owl 
Eorm  using  MS  ErontPage  2002.  Portions  of  each  of  these  templates  are  shown  in  Eigure 
17  and  Eigure  18.  The  forms  in  their  entirety  are  shown  in  Appendix  B.  MS  ErontPage 
2002  allowed  is  to  save  the  results  of  the  survey  to  a  database  on  the  server.  The  two 
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databases  based  on  these  two  surveys,  Demographic  and  Lark  and  Owl  databases,  would 
accumulate  the  survey  results  from  each  participant’s  submission  of  the  data.  Because 
the  storage  database  is  also  on  the  server,  we  were  able  to  update  this  survey  information 
into  the  main  Study  Repository  Template  Database  with  a  simple  update  query  function 
added  to  the  start  up  of  the  database. 


He 


Naval  Po.stgraduate  School 

Ho-lf  >v  Ca  I  crn  j 


Local  Demographic  Survey 
Instructrons: 

Please  fill  out  ail  likxks  in  this  survey,  when  Hnished  please  press 
the  "SU6MIT"  button 


1. NAME:  Latl;  |  FIrtt: 

2.  Rank;  I0-3J 

3.  Service:  |'JSN  J  Prtw Service  P  Tes 

4.  E'likoM 

5.  Homr*  Phone: 

fS5S;5SS'&5£-5 

6.  PouHi'PNUttilier/Cefl  [n^  " 

7.  Horns  Addr^»sJ 

C«vJ 


Slots: 


8  4M»e: 

0.  Condor:  ^  *^010  ^  Fomalo 

10.  HDight;  H  ^footlO  ^Inches  (round  to  tho  naara«t  incii) 

11.  Wolqllt;|  ibe 

12.  Oo  yon  woke  up  lo  on  olorm  Hoc*?  ^  Ye»  ^  Ka 


Figure  17.  Demographic  Survey  Template  Form 
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Lark  and  Owl 

Instructions: 

Take  the  following  survey  to  help  us  determine  if  you  are  a  lark,  and  owl  or  somewhere  in 
between.  These  questions  should  be  answered  as  if  you  were  at  home  and  not  deployed? 

Last  Name  I  Type  your  last  name 

Circadian  Identity: 

1.  During  your  last  vacation  week,  how  often  did  you  get  up  later  than  planned  or  have 
difficulty  getting  ready  on  time  even  though  you  went  to  bed  at  your  regular  time? 

^  It  never  happened 

^  It  happened  once 

^  It  happened  two  or  more  times 

^  It  happened  more  than  three  times 

2.  When  you  have  no  commitments  the  next  day,  at  what  time  do  you  go  to  bed  compared 
with  your  usual  time? 

^  Seldom  or  never  later 

Figure  18.  Lark  and  Owl  Template  Form 

As  indicated  earlier,  if  the  web  is  not  aeeessible  to  the  participants,  copies 
of  these  forms  (shown  in  Appendix  D)  are  ineluded  in  the  MS  Aecess  Study  Repository 
Tenplate  Database. 

b.  Log  Database  Design 

After  careful  investigation  of  existing  PDA  logging  programs,  no  logging 
software  for  PDAs  was  available  to  reeord  the  types  of  events  SMEs  had  artieulated  for 
the  daily  event  log.  Though  the  log  was  easy  to  design  using  MS  Aeeess,  there  was  no 
way  to  export  this  desktop  version  to  an  applieation  available  for  the  PDAs.  The  NPS  IT 
staff  does  not  eurrently  have  a  site  license  for  any  PDA  database  development  software. 
Programming  applications  for  PDA  use  was  beyond  the  seope  of  our  expertise.  After 
evaluating  several  packages  available  on  the  commercial  market  as  well  as  freeware,  we 
chose  HanDBase  3.0  for  our  PDA  database  development.  This  program  provides  the 
capability  to  create  and  modify  databases  from  both  a  desktop  and  PDA  platform. 
HanDBase  offered  enough  funetionality  and  teehnical  support  to  meet  study 
requirements.  We  created  a  simple  log  database  template  using  HanDBase  software 
which  is  fairly  easy  to  modify.  Because  no  site  licensing  was  available  for  HanDBase 
3.0,  a  license  is  required  for  every  PDA. 
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L*g  Event 


ID#  Kondy 
Stili-n! 


1.  Date:  No  Date 
I.  Time:  No  Time 
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Time 
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r/b/os 

1’45 

OFf  Dut' 

V7.^03 

0  0^ 

Down  foe 
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6  20 

WoEe  Up 

i/7/o: 

620 

VV(r»-:h  Off 

3/7/03 

6-19 

Watch  ON 

3/'?,-03 

750 

*  n  [  y . 

3/7/0'; 

17  15 

OFF  Du* ; 

yim 

22  43 

Down  for 

3/3/03 

551 

WoF  e  Up 

a  □  ^  E 

HOM*  Tbolr  Fifv^  'T 

Figure  19.  PDA  Event  Log  Tempiate 


Loaded  on  each  PDA,  the  log  database  enables  a  participant  to  record  an 
event,  the  date  and  time  of  the  event  and  a  short  note  if  clarification  is  required.  The 
events  we  included  in  the  template  are  generic  and  can  easily  be  modified  to  include 
particular  events  the  researchers  would  like  to  capture  for  a  given  study.  A  screen 
capture  from  the  PDA  of  the  Event  Log  Template  with  sample  data  is  shown  in  Figure 
19.  The  recorded  data  are  stored  on  a  file  in  the  PDA  and  can  be  downloaded  to  a  PC 
using  the  HotSync  function  of  the  PDA.  The  HanDBase  software  includes  a  desktop 
version  allowing  the  data  from  each  PDA  to  be  easily  consolidated  and  exported  to  a  MS 
Excel  spreadsheet. 

c.  Collecting  ARES  (ANAA^^  Readiness  Evaluation  System)  Data 
A  collection  means  for  performance  data  is  available  on  the  PDA  using 
ARES  software  shown  in  Figure  20.  Researchers  generally  require  participants  to 
complete  a  designated  ARES  battery  of  tests  (e.g.,  reaction  time,  running  memory,  and 
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sleepiness  scale)  at  least  three  times  a  day.  Data  are  recorded  for  each  test  and  are  stored 
into  a  PDA  database  file.  When  the  PDA  performs  a  HotSync  function  with  the  laptop, 
these 


AKIS 


IM 


AMAM 


RRES  is  a  series  of  simple  tftsts 
of  mental  fivKtion. 


Log  In  (295  days  left) 


Figure  20.  ARES  Software  for  the  PDA 


d.  Research  Team  Input  Forms 

Data  are  saved  to  a  database  file  which  is  able  to  be  read  only  by  the  Data  Manager 
software. 

Figure  21  shows  the  raw  data  in  Data  Manager  software.  From  Data 
Manager,  the  researcher  can  use  the  copy  function  to  copy  the  data  to  a  MS  Excel 
spreadsheet  for  analysis  (shown  in  Figure  22). 
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Figure  21.  Raw  ARES  Data  in  Data  Manager  Software 
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Figure  22.  Raw  ARES  Data  Copied  to  MS  Access  Database 


Other  data  maintained  by  research  teams  will  also  be  collected 
electronically.  Information  on  the  research  team  and  study  parameters  will  be  recorded 
by  the  researchers  using  automated  input  forms  shown  in  Figure  23  and  Figure  24 
respectively.  Additionally,  researchers  will  maintain  and  update  equipment  statuses, 
assign  equipment  to  participants  and  record  status  of  collection  of  data  and  equipment 
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from  participants  at  the  end  of  a  study  using  simple  input  forms.  All  input  forms 
available  in  the  Study  Repository  Template  Database  are  also  shown  in  Appendix  D.  The 
use  of  these  electronic  forms  increases  the  efficiency  of  the  research  team’s  ability  to 
conduct  the  study  and  minimizes  human  errors  associated  with  tracking  by  pen  and  paper. 


Figure  23.  Input  Research  Team  Information  Form 
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Figure  24.  Input  Study  Information  Form 


2.  Database  Design 

We  chose  Microsoft  Access  2002  as  the  software  to  create  our  Study  Repository 
Template  Database  because  it  provided  adequate  support  for  small  relational  database 
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development.  It  is  fairly  easy  to  use  and  it  is  supported  by  NPS  IT  staff.  There  were  no 
additional  licensing  requirements  to  use  it  and  there  is  a  fairly  extensive  MS  Access 
knowledge  base  at  NPS  to  assist  research  teams  with  database  modifications.  We 
examined  the  feasibility  of  using  the  latest  version  of  Extensible  Markup  Language 
(XML)  developed  and  supported  by  the  World  Wide  Web  Consortium  (W3C)  to  create 
the  database  to  avoid  proprietary  issues  and  created  a  prototype  database  using  XML  Spy 
software.  While  there  are  many  benefits  to  using  XML,  the  current  W3C  version  of 
XML  does  not  easily  support  relational  databases  and  the  complexity  involved  in 
maintaining  referential  integrity  for  our  database  proved  too  complex.  According  to  the 
W3C  discussion  groups,  future  versions  of  XML  will  provide  more  capabilities  to 
develop  relational  databases.  Microsoft  has  created  a  version  of  XML  (MSXML)  which 
supports  relational  database  development;  however,  MSXML  is  not  W3C  compliant  and 
is  clearly  proprietary.  Additionally,  MSXML  is  not  exclusively  a  database  development 
tool.  Creating  databases  using  MSXML  requires  extensive  knowledge  of  the  XML 
language  as  well  as  a  thorough  understanding  of  databases.  Because  of  its  maturity  as  a 
product,  MS  Access  2002  proved  to  be  a  more  robust  solution  for  us  to  use.  If  future 
W3C  versions  of  XML  support  relational  databases  adequately,  the  Study  Repository 
Template  database  we  have  created  can  be  exported  to  a  XML  format  for  modification 
and  maintained  as  an  XML  database  for  suture  use. 

Transforming  entities  to  relations,  we  recognized  the  attributes  identifying  each 
entity  and  made  these  the  primary  keys  of  the  relations.  When  any  of  these  primary  keys 
were  required  as  attribute  of  other  relations,  we  depicted  them  as  foreign  keys.  Ligure  25 
depicts  the  relationships  created  using  MS  Access  2002.  The  highlighted  text  in  each 
relation  indicates  the  primary  key(s). 
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Figure  25.  Study  Repository  Template  Relationships 


3.  Application  Design 

The  Study  Repository  Template  Database  was  designed  to  provide  research  teams 
with  a  robust  database  to  which  minor  modificatbns  can  be  made  to  tailor  it  for  a  given 
study.  In  designing  our  database  application,  no  mechanisms  for  limiting  control  were 
necessary  because  the  research  team  will  be  the  administrators  of  their  modified 
databases.  In  this  template  design,  input  forms  and  reports  were  created  to  allow  the 
research  team  to  input  data  efficiently  and  retrieve  data  in  useful  formats  for  data 
analysis.  A  Main  Menu  Form  was  created  to  allow  researchers  to  easily  navigate  their 
way  through  these  useful  forms  and  reports. 

E.  IMPLEMENTATION  PHASE 

1.  Creating  the  Database 

We  published  the  template  web  surveys,  created  the  event  log  on  the  PDA,  loaded 
ARES  on  the  PDA  and  created  the  research  team  input  forms  according  to  the  design. 
After  these  data  collection  methods  were  refined,  we  were  able  to  finalize  the  Study 
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Repository  Template  Database  design  and  create  it  using  Microsoft  Access  2002.  A 
listing  generated  by  the  Access  program  of  the  database  tables,  their  properties,  and 
relationships  can  be  found  in  Appendix  C.  In  Appendix  D,  graphical  representations  of 
all  forms  and  reports  available  from  the  Main  Menu  Form  of  the  Study  Repository 
Template  Database  are  also  available.  This  database  resides  on  the  HSIL  web  server 
along  with  the  web  survey  forms.  The  Event  Log  and  the  ARES  program  are  loaded  on 
PDAs  in  the  HSIL  and  the  accompanying  software  to  both  of  these  programs  is  loaded  on 
the  desktops  in  the  HSIL  as  well  as  the  laptop  that  accompanies  the  research  team  to  the 
location  of  the  studies  conducted. 

2.  Storing  Data  to  the  Database 

At  the  start  of  a  study,  the  research  team  can  enter  data  to  the  RESEARCH  TEAM 
and  STUDY  tables  using  the  forms  from  the  Main  Menu  Eorm  shown  in  Eigure  26. 
These  forms  are  also  displayed  in  Appendix  D.  Researchers  can  also  update  the  status  of 
the  Actigraphs  and  the  PDAs  in  the  ACTIGRAPH  STATUS  and  PDA  STATUS  tables 
respectively,  using  input  forms  available  from  the  same  Main  Menu  Eorm. 
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Figure  26.  Main  Menu  Form 

53 


After  the  participants  have  been  identified  and  have  completed  the  online  surveys, 
the  Study  Repository  Template  Database  automatically  updates  the  PARTICIPANT  and 
LARK  AND  OWL  tables  when  the  repository  study  database  is  opened  with  the  data  that 
has  been  entered  by  participants  via  the  web  and  saved  to  the  Demographic  and  the  Lark 
and  Owl  databases.  If  web  access  is  not  available  to  the  participants,  the  research  team 
can  allow  the  participants  to  complete  the  input  forms  created  with  MS  Access  available 
from  the  Main  Menu  Form.  If  the  participants  have  no  access  to  computers,  these  input 
forms  can  even  be  printed  if  necessary  and  the  researcher  can  add  the  data  to  the  database 
using  the  completed  printed  forms  and  the  database  input  form  to  ease  the  transfer  of  data 
from  paper  to  the  database.  The  researchers  can  update  the  ACTIGRAPH  and  PDA 
tables  using  the  Assign  Actigraph  and  Assign  PDA  Forms  respectively,  shown  in 
APPENDIX  D.  After  the  research  team  downloads  the  ARES  and  EOG  data  from  PDAs 
to  MS  Excel  spreadsheets,  the  datasheets  are  copied  to  the  respective  tables.  Einally,  as 
the  research  team  collects  equipment,  they  can  use  the  Collect  Equipment  Eorm  to  update 
the  COEEECT  EQUIPMENT  AND  DATA  table. 

Various  reports  are  available  to  the  research  team  to  use  during  or  after  the  study 
and  are  also  accessible  from  the  Main  Menu  Eorm.  Examples  of  all  of  these  reports 
available  are  shown  in  Appendix  D.  These  were  designed  to  assist  the  researchers  with 
assigning  and  keeping  track  of  equipment  used  in  the  study  as  well  as  to  print  the  final 
event  log.  This  Eog  Report,  shown  in  Eigure  27,  includes  conditional  formatting  which 
allows  the  research  team  to  easily  code  the  actigraphy  data  and  correlate  the  participants’ 
effectiveness  with  their  daily  activities  using  EAST  and  ARES  analysis  tools. 

3.  Application  Implementation 

Though  the  research  team  will  modify  the  design  template  to  suit  the  individual 
study  requirements,  the  Main  Menu  Eorm  allows  the  researcher  to  easily  navigate 
through  the  database  updating  information  and  generating  valuable  reports  as  needed. 
This  Main  Menu  Eorm  can  also  be  modified  to  accommodate  the  database  changes  made 
for  a  particular  study  or  to  add  more  useful  reports  required  for  the  research  being 
conducted. 
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Figure  27.  Log  Event  Report 


When  this  template  design  was  created,  we  assumed  participants  would  have 
access  to  the  web  to  provide  survey  information.  If  this  is  not  possible,  researchers  can 
recreate  these  surveys  using  MS  Access  forms.  When  the  system  is  used  in  this  capacity, 
it  is  important  for  the  research  team  to  incorporated  adequate  control  mechanisms  to 
allow  participants  to  enter  and  verify  data  while  preventing  them  from  accessing 
restricted  data.  The  Help  feature  in  MS  Access  outlines  these  control  features  in  a  “how¬ 
to”  format  that  is  easy  for  the  researchers  to  follow.  For  the  forms  created  using  MS 
Access  in  the  database  template,  constraints  were  used  to  reduce  the  potential  for  data 
entry  errors  for  the  research  team.  Many  forms  include  drop  down  lists  and  check  boxes 
to  prevent  ambiguity  or  misrepresented  data.  An  example  of  this  is  the  Assign 
Actigraphs  Form  shown  in  Figure  28.  The  drop  down  list  for  the  Actigraph  Number  is 
taken  from  the  query  based  on  the  ACTIGRAPH  STATUS  Table.  This  not  only  ensures 
researchers  label  the  numbers  with  the  same  format  but  also  eliminates  the  error  of 
assigning  an  Actigraph  that  has  not  been  added  to  the  ACTIGRAPH  STATUS  Table. 

Similarly,  the  dropdown  list  for  the  Participant’s  Last  Name  is  available  from  a  query 
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based  on  the  PARTICIPANT  table  allowing  the  researcher  to  only  assign  Actigraphs  to 
identified  participants.  Also,  once  the  name  is  selected,  the  Participant  ID  field  is 
automatically  filled  with  the  Participant  ID  associated  with  the  selected  name. 


Figure  28.  Assign  Actigraph  Input  Form 

F.  EVALUATING  THE  REENGINEERED  PROCESS 

The  entire  application  including  the  web  Demographic  and  the  Lark  and  Owl 
Surveys,  the  PDA  Event  Log,  the  ARES  Program  for  the  PDA,  the  Study  Repository 
Template  Database  with  research  input  forms  along  with  the  Actigraphs  was  packaged 
and  tested.  A  group  of  volunteer  students  at  NPS  were  modeled  as  a  group  of  study 
participants.  As  the  simulated  research  team,  we  asked  them  to  take  the  web  surveys, 
wear  the  Actigraph  watches,  take  the  ARES  battery  of  tests  on  a  set  schedule  and  log 
their  daily  events  with  the  PDA  for  a  period  of  a  week.  We  recorded  study  and  research 
team  information,  updated  the  equipment  statuses,  and  assigned  equipment  to  the 
participants  after  they  had  completed  the  web  surveys.  At  the  conclusion  of  the  week,  we 
collected  the  equipment  and  downloaded  the  data  from  the  Actigraphs  and  PDAs.  We 
copied  the  ARES  and  Log  Data  to  the  database  and  conducted  exit  interviews  with  the 
participants.  Making  no  significant  modifications,  we  were  satisfied  with  the  prototype 
of  the  “TO-BE”  process. 
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V.  REENGINEERED  PROCESS 


A.  OVERVIEW 

The  newly  created  IT  tools  described  in  Chapter  IV  were  incorporated  into  the 
reengineered  “AS-IS”  process.  In  this  chapter,  the  new  “TO-BE”  process  is  outlined. 
Using  the  seven  step  KVA  process,  the  “TO-BE”  process  Return  on  Knowledge  (ROK) 
was  calculated  at  the  process  and  sub -process  levels  using  very  conservative  estimates  to 
provide  the  means  for  comparison  of  the  “AS-IS”  to  the  “TO-BE”  process.  A  report  of 
the  evaluation  of  this  KVA  comparative  analysis  is  also  provided.  At  the  conclusion  of 
this  chapter,  a  discussion  of  the  potential  for  a  dramatic  increase  to  ROK  through 
implementation  of  a  radical  redesign  to  the  process  is  offered.  This  impact  of  this  radical 
redesign  for  the  Department  of  the  Navy  and  possibly  other  services  in  the  Department  of 
Defense  is  worthy  of  consideration. 

B.  “TO-BE”  PROCESS  MODEL 

As  we  conducted  the  test  in  the  laboratory  using  the  volunteer  students  as 
participants,  we  carefully  reengineered  the  “AS-IS”  process.  Incorporating  the  newly 
designed  data  collection  techniques  and  the  database  template,  we  were  able  to  model  the 
‘TO-BE”  fatigue  study  process.  Applying  the  KVA  methodology,  we  determined  the 
core  and  sub -processes  for  the  “TO-BE”  process.  Though  the  titles  of  these  remain  the 
same  as  in  the  “AS-IS”  process,  many  of  the  subordinate  activities  and  tasks  changed.  A 
complete  set  of  diagrams  of  these  sub -processes,  activities  and  tasks  is  provided  with 
KVA  calculations  in  APPENDIX  A.  The  following  paragraphs  highlight  the  significant 
changes  made  and  the  impact  they  have  for  the  efficiency  and  accuracy  of  the  data 
collection  and  analysis  conducted  by  research  teams  from  the  HSIE  at  NPS. 

1.  Phase  1:  Collect  Data  Sub- Process 

The  major  reengineering  effort  happened  in  this  phase.  Eigure  29  depicts  the 
Collect  Data  Sub-Process  of  the  “AS-IS”  process  as  the  baseline  for  comparison  and 
Eigure  30  shows  the  Redesigned  Collect  Data  Sub-Process  of  the  “TO-BE”  process.  The 
Prepare  Data  for  Collection  Activity  now  includes  modifying  the  Study  Repository 
Template  Database,  modifying  the  Demographic  Survey  on  the  HSIE  webpage, 
appending  the  Eark  and  Owl  Survey  to  the  webpage,  and  modifying  the  PDA  Event  Eog. 
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Figure  29.  AS-IS  Collect  Data  Sub- Process 


http://www.nps.navv.mil/humanfactors.  The  web  surveys,  their 
corresponding  databases  and  the  Study  Repository  database  are  all  stored  on  the  server. 
These  administrative  tasks  take  considerably  less  time  than  those  under  the  corresponding 
“AS-IS”  activity  because  the  researchers  modify  existing  templates  rather  than  create 
them  on  their  own.  Additionally,  by  completing  these  tasks  at  this  point  in  the  process, 
the  research  team  is  prepared  to  collect  survey  data  via  the  HSIL  webpage  once 
participants  have  been  identified  by  the  sponsoring  organization  of  the  study.  This 
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participant  survey  data  are  also  checked  for  completeness  and  screened  for  potential 
errors  or  issues.  This  pre-screening  of  participant  data  allows  researchers  the  ability  to 
prevent  the  20%  loss  of  data  generally  experienced  using  the  “AS -IS”  process. 

Once  the  participants  are  recorded  in  the  database,  the  equipment  is  prepared. 
The  Actigraph  watches  are  initialized  and  assigned  by  serial  number  to  the  participants 
using  the  Assign  Actigraph  Form.  The  PDAs  are  similarly  assigned  by  name  (e.g., 
HSILl,  HSIL2  etc.)  to  participants  using  the  Assign  PDA  Form.  The  HanDBase 
software  and  the  Study  Event  Log  database  file  and  form  file  are  loaded  onto  the  PDA. 
Additionally,  the  ARES  software  is  loaded  onto  the  PDAs  and  each  participant  is 
registered  by  participant  ID  and  the  particular  battery  of  tests  is  selected.  Consent  forms 
are  printed  and  all  equipment  and  forms  are  bundled  for  issue  under  the  Prepare 
Equipment  Activity.  This  activity  can  now  be  completed  prior  to  the  research  team  ever 
arriving  at  the  study  location.  A  tremendous  burden  is  lifted  from  the  researchers  by 
doing  so.  Now  they  can  come  to  the  study  location  and  focus  on  collecting  accurate 
actigraphy,  ARES  performance  data  and  log  data  instead  of  worrying  about  the  logistics 
of  identifying  participants  and  tracking  the  assignment  and  issue  of  equipment  to  them. 
During  the  collection  process,  researchers  can  check  participant  data  for  completeness  by 
routinely  downloading  data  from  the  Actigraphs  and  the  PDAs.  This  ensures  that 
equipment  is  working  properly,  participants  are  taking  their  ARES  tests,  and  that  they  are 
adequately  logging  their  daily  behavior.  At  the  end  of  this  sub -process,  the  research  team 
will  have  a  record  of  the  study  parameters  and  the  research  team  information  as  well  as 
“clean”  participant  and  Lark  and  Owl  data  loaded  in  their  database.  They  will  have 
PDAs  and  Actigraphs  with  data  stored  on  them  and  signed  participant  consent  forms. 
Compared  with  the  “AS-IS”  sub-process,  the  research  team  will  have  collected  more 
accurate  data  and  will  have  already  completed  all  of  the  survey  data  preparation  for 
analysis  by  the  end  of  this  phase  in  the  “TO-BE”  process. 

2.  Phase  2:  Prepare  Data  for  Analysis  Sub -Process 

Comparing  the  “AS-IS”  Prepare  Data  for  Analysis  Sub -Process  shown  in  Eigure 
31  to  the  “TO-BE”  Sub-Process  shown  in  Eigure  32,  the  Input  Data  to  Excel 
Spreadsheets  Activity  and  the  Clean  Data  Activity  are  eliminated.  Because  the 
Demographic  and  Lark  and  Owl  survey  data  have  been  collected  and  verified  for 
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completeness  in  the  Collect  Data  Sub-Process,  the  only  data  requiring  preparation  for 
analysis  in  the  ‘TO-BE”  Prepare  Data  for  Analysis  Sub-Process  would  be  the  Event  Eog 
data,  the  Actigraph  data,  and  the  newly  added  ARES  data. 
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loads  the  ARES  Data  (.PDB  file)  and  copies  the  data  to  the  ARES  Data  table  in  the  Study 
Repository  database.  The  research  team  can  use  the  Data  Manager  software  to  conduct 
analysis  and  the  data  in  the  MS  Access  table  can  be  exported  easily  to  other  data  analysis 
software.  The  Event  Eog  Data  are  viewed  with  the  HanDBase  Desktop  software  and 
copied  to  the  Eog  table  in  the  Study  Repository  database.  Without  further  manipulation, 
the  Eog  Report  is  generated  by  clicking  the  “Eog  Report”  command  button  on  the  Main 
Menu  Eorm  in  the  database.  The  conditionally  formatted  Eog  Report  highlights  the 
“ON/OEE”  state  of  an  event,  for  example  Actigraph  Watch  ON  and  Actigraph  Watch 
OEE.  The  report  is  generated  in  chronological  order  by  participant.  On  the  web  server,  a 
Study  Repository  folder 

At  the  heart  of  fatigue  studies  are  the  actigraphy  data.  The  scoring  algorithm  used 
in  the  Actigraph  software  determines  whether  a  sleep  epoch  is  classified  as  either  sleep  or 
wake.  Based  on  a  participant’s  log  and  a  researcher’s  subjective  expertise,  the  Actigraph 
data  nust  be  “cleaned.”  If  the  software  coded  low  activity  levels  as  sleep  when  the 
participant  was  really  awake,  the  participant  may  be  wrongfully  thought  to  have  had  more 
sleep  and  thus  potential  higher  performance  effectiveness  than  he  actually  had.  To 
accurately  portray  a  participant’s  activity,  the  sleep/wake  log  is  essential.  Eigure  33 
shows  a  participant’s  Actigraph  data  over  a  period  of  a  few  days.  The  sleep  algorithm  in 
the  Action  W  Software  scored  the  low  activity  levels  in  the  circled  area  as  sleep.  The 
intermittent  activity  shown  during  this  time  could  lead  a  researcher  to  concur  with  the 
Actigraph  software  and  assume  the  participant  was  asleep.  In  this  case,  however,  the 
accompanying  log  report,  shown  in  Eigure  34,  clearly  indicates  that  during  that  time,  the 
participant  had  removed  his  Actigraph  watch  but  was  not  asleep.  By  recoding  the  sleep 
data  as  wake  data,  the  researcher  is  able  to  more  accurately  represent  the  behavior  of  that 
participant.  Because  of  the  format  of  the  Eog  Report,  it  is  easy  to  see  the  changes  of  state 
a  participant  goes  through  chronologically.  The  highlighted  events  allow  the  research 
team  to  easily  compare  the  recorded  actigraphy  with  the  log  report  to  trim  the  data.  This 
ease  of  this  comparison  not  only  saves  researchers  many  hours  of  trimming  data  but  it 
also  yields  a  more  accurate  assessment  of  the  participant’s  activity  than  the  “AS -IS” 
process  does  using  paper  logs.  Because  the  Actigraph  data  files  can  only  be  read  by 
Action  W  and  EAST  software  in  particular  data  formats  (.AMI  and  .EBE  respectively),  it 
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is  not  useful  to  translate  these  files  into  MS  Access  formats.  Instead,  the  Actigraph  and 
FAST  files  (both  raw  and  “cleaned”)  are  saved  to  the  Study  Repository  folder  on  the 
server. 


Tue  4/1/2003 
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Figure  34.  Log  Report  Data  for  Participant  01 


The  most  significant  change  to  the  second  phase  of  the  process  is  that  the 
researchers  do  not  have  to  transpose  data  from  paper  to  electronic  format.  Not  only  does 
this  save  the  time  of  entering  the  data  into  spreadsheets  but  more  significantly,  it  also 
eliminates  the  countless  hours  spent  comparing  the  spreadsheets  with  the  original  paper 
entries  to  verify  the  data  accuracy.  Secondly,  another  tremendous  benefit  of  this 
redesigned  sub -process  is  the  amount  of  time  saved  in  “cleaning”  the  Actigraph  data 
using  the  Log  Report  generated  from  the  raw  Event  Log  data.  Next,  the  standardized 
format  included  with  the  data  collection  and  storage  templates  decreases  the  amount  of 
time  researchers  spend  formatting  and  storing  their  data.  Finally,  the  addition  of  the 
ARES  data  provides  a  valuable  performance  measure  for  the  research  team  while 
requiring  minimal  time  to  download  the  data  using  the  HotSync  function  and  to  copy  it  to 
the  MS  Access  database.  In  total,  this  “TO-BE”  Redesigned  Prepare  Data  for  Analysis 
Sub-Process  will  take  less  than  half  the  cycle  time  to  complete  than  the  “AS-IS”  Prepare 
Data  for  Analysis  Sub -Process  while  providing  the  research  team  with  more  accurate  data 
to  analyze. 
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3.  Phase  3:  Analyze  and  Publish  Sub- Process 

The  data  analysis  piece  of  the  third  phase  of  the  “TO-BE”  fatigue  study  process  is 
very  similar  to  that  of  the  ‘AS-IS”  process.  The  thesis  output  of  the  sub -process  is  the 
same.  The  major  difference  is  the  inclusion  of  the  Study  Repository  database  as  the 
means  of  data  storage  available  for  reuse  in  the  “TO-BE”  process.  Eigure  35  shows  the 
“AS-IS”  Analyze  and  Publish  Sub-Process.  The  “TO-BE”  Redesigned  Analyze  Data, 
Publish  Analysis  and  Store  Study  Data  Sub-Process  depicted  in  Eigure  36  shows  the 
addition  of  Clean  MS  Access  Database  to  the  Write  Thesis  Activity  resulting  in  the  new 
Study  Repository  folder  including  the  database  and  complimentary  data  files  as  an  output 
product  of  the  sub-process. 


Analyze  and 
Publish 


Figure  35.  AS-IS  Analyze  and  Publish  Sub-Process 
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analysis  and  does  not  have  to  waste  time  designing  a  storage  method.  The  Study 
Repository  database  allows  them  the  option  to  add  tables  for  analyzed  data  results  and  to 
add  or  modify  reports  for  inclusion  in  the  published  documentation  of  the  study.  MS 
Access  empowers  the  researcher  with  flexibility  to  efficiently  utilize  the  collected  data. 
Additionally,  the  standard  format  allows  those  interested  in  the  fatigue  research  of  the 
HSIL  at  NFS  to  more  easily  access  and  interpret  the  collected,  analyzed  and  stored  data 
of  the  operational  fatigue  studies. 

C.  RETURN  ON  KNOWLEDGE  FOR  “TO-BE”  PROCESS  USING  KVA 
ANALYSIS 

To  calculate  a  ROK  (Return  on  Knowledge)  of  the  newly  redesigned  sub- 
processes  in  order  to  compare  it  with  the  current  process,  we  used  the  seven  step  KVA 
methodology.  A  summary  of  the  reengineered  “TO-BE”  core  process  and  sub-processes 
identified  in  STEP  1  are  shown  in  Eigure  37. 
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We  kept  the  same  units  of  hours  to  calculate  learning  time  as  we  did  in  STEP  2 
for  the  “AS-IS”  analysis.  In  STEP  3,  the  learning  time  for  each  task  was  calculated.  The 
sampling  time  period  of  one  year  remained  the  same  as  was  used  for  the  “AS-IS”  for 
STEP  4.  In  STEP  5,  we  calculated  the  weighted  learning  time  for  each  sub-process.  Eor 
the  tasks  that  remained  the  same,  we  carried  their  learning  times  from  the  “AS-IS” 
process.  Eor  the  newly  incorporated  tasks,  we  used  the  expertise  of  the  SMEs  and  our 
own  to  determine  the  learning  time.  Because  of  the  dramatic  reduction  in  cycle  time  for 
the  entire  process,  we  conservatively  estimated  that  the  number  of  studies  conducted  each 
year  could  easily  double  in  number.  Additionally,  the  less  obtrusive  means  of  collection 
will  most  likely  allow  for  an  increase  in  the  average  number  of  participants  per  study.  As 
a  conservative  measure,  we  kept  25  as  the  average  number  of  participants  per  study  for 
our  calculations.  However,  due  to  the  streamlined  efficiency  of  collecting  and  cleaning 
survey  data  from  participants  prior  to  arriving  on  site,  we  maintain  that  the  research  team 
will  be  able  to  collect  useable  data  on  all  25  participants.  When  calculating  the  weighted 
learning  time  of  a  task,  we  multiplied  the  learning  time  of  a  task  (included  associated  IT 
learning  time  if  applicable)  with  the  number  of  times  the  task  was  executed  in  a  given 
sample  period.  Since  the  average  number  of  participants  is  25  and  the  number  of  studies 
conducted  per  year  increased  to  two,  the  weighted  learning  time  for  a  task  such  as 
initializing  Actigraphs  would  be  the  total  learning  time  multiplied  by  50.  Cumulative 
weighted  learning  times  were  calculated  for  the  process  and  sub-process  level.  Similarly 
in  STEP  6,  the  cost  to  execute  each  task  was  calculated  as  well  as  the  cumulative  costs  of 
each  sub-process.  To  determine  the  ROK  according  to  STEP  7,  the  ratio  of  weighted 
learning  time  divided  by  cost  was  calculated.  A  comprehensive  table  of  calculations  by 
task  is  included  in  Appendix  A. 

A  comparative  summary  of  the  calculated  ROK  between  the  “AS-IS”  process  and 
the  ‘TO-BE”  process  is  shown  in  Table  2.  Though  the  estimations  were  conservative,  the 
ROK  of  the  “TO-BE”  process  more  than  doubles  that  of  the  “AS-IS”  process.  The 
efficient  data  collection  means  of  the  “TO-BE”  process  clearly  enables  the  research  teams 
to  analyze  more  data  in  a  timely  and  efficient  manner  producing  more  accurate  results. 

The  bottoms-up  approach  we  used  of  evaluating  KVA  from  the  task  level  to  the 

entire  process  captured  with  granularity  the  increased  ROK. 
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Sub  Processes 

Weighted 
Learning  Time 
(Hrs) 

Total  Cost 

($) 

As -Is 

ROK 

To-Be 

ROK 

Collect  Data 

289 

3300 

9% 

1810 

4246 

43% 

Prepare  for  Analysis 

1158 

8278 

14% 

3075 

9185 

33% 

Analyze  and  Publish 

1161 

10192 

11% 

6885 

32082 

21% 

TOTAL 

2608 

21770 

12% 

11770 

45513 

26% 

Table  2.  Comparison  of  “AS  -IS”  ROK  to  “TO-BE”  ROK 


Additional  benefits  of  the  reengineered  proeess  that  have  yet  to  be  realized. 
Perhaps  the  most  notable  of  these  is  the  addition  of  the  data  repository  for  all  the  HSIL 
studies  eondueted  at  NPS  whieh  will  enable  researehers  to  eonduet  meta-analysis  aeross 
many  studies.  Additionally,  the  standardization  of  the  “TO-BE”  proeess  and  the  ability 
to  save  methods  used  for  eaeh  study  eondueted  may  allow  an  even  greater  number  of 
studies  to  be  eondueted  per  year  the  pool  of  resourees  available  to  thesis  student  fatigue 
researehers  beeomes  rieher  with  every  eompleted  study. 

D.  IMPACT  OF  A  DRAMATIC  PROCESS  REENGINEERING  EFFORT 

The  goal  of  the  fatigue  researeh  eondueted  today  and  in  the  near-term  future  at  the 
HSIL  at  NPS  is  to  evaluate  partieipant  data  and  provide  feedbaek  to  eommanders  on  the 
effeets  of  fatigue  as  it  impaets  human  performanee  effeetiveness.  Migrating  from  the 
“AS-IS”  proeess  to  the  “TO-BE”  proeess  will  eertainly  allow  researehers  to  aehieve  this 
goal  more  effieiently  and  with  greater  volume  potentially  allowing  the  ability  to  generate 
more  representative  data.  However,  what  if  the  goal  was  expanded  to  provide 
eommanders  with  near  real  time  assessments  of  fatigue  and  performance  effectiveness  to 
allow  them  to  make  smarter  decisions? 

By  further  dramatic  process  reengineering,  this  goal  could  be  realized.  Instead  of 
focusing  research  on  determining  past  performance  effectiveness,  the  HSIL  at  NPS  could 
provide  commanders  with  more  predictive  tools  to  assess  current  conditions  and  risks 
associated  with  fatigue  and  provide  the  commanders  with  strategies  to  mitigate  given 
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circumstances  to  increase  this  effectiveness  for  a  particular  mission.  By  packaging  the 
data  collection  process  into  “How-To”  type  fatigue  assessment  kits  issued  to  each  ship, 
sailors  and  other  service  members  could  be  assessed  in  their  operational  environments. 
Data  collected  aboard  the  ship  by  trained  members  of  the  crew  could  be  sent 
electronically  to  the  team  of  researchers  at  the  HSIL  where  they  would  conduct  their 
analysis  and  provide  feedback  to  the  commanders.  A  rough  estimate  of  the  ROK  for  the 
Dramatic  Redesign  was  calculated  and  is  shown  in  Appendix  A.  A  summary  of  the  ROK 
results  compared  with  the  “AS-IS”  and  “TO-BE”  processes  are  shown  in  Table  3. 


Sub  Processes 

Weighted 

Learning 

Time 

Total 

Cost 

As -Is 

ROK 

To-Be 

ROK 

Dramatic 

ROK 

Collect  Data 

289 

3300 

9% 

1810 

4246 

43% 

395760 

370425 

107% 

Prepare  for  Analysis 

1158 

8278 

14% 

3075 

9185 

33% 

1204500 

2062500 

58% 

Analyze  and  Publish 

1161 

10192 

11% 

6885 

32082 

21% 

254250 

1122413 

23% 

TOTAL 

2608 

21770 

12% 

11770 

45513 

26% 

1854510 

3555338 

52% 

Table  3.  Comparison  of  “Dramatic”  ROK  to  “AS-IS”  and  “TO-BE”  ROK 


A  repository  of  successfully  mitigated  obstacles  could  be  retained  and  shared  as  a 
“Lessons  Learned”  type  of  tool  for  leaders  to  use.  The  staff  of  researchers  at  the  HSIL 
would  certainly  have  to  increase  and  could  not  be  limited  to  thesis  students  but  the 
incredible  return  provided  to  the  entire  Navy  could  positively  impact  the  success  of  future 
missions  and  reduce  number  of  accidents  and  other  incidents  caused  by  human  error  due 
to  fatigue. 
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VI.  NAVAL  OFFICER  INDOCTRINATION  SCHOOL  STUDY 


A.  OVERVIEW 

The  reengineered  proeess  was  validated  using  partieipants  from  the  Naval  Offieer 
Indoctrination  School  (OIS)  in  Newport,  RI  (Figure  38).  To  accomplish  the  validation, 
we  designed  a  study  to  emulate  the  NFS  Fatigue  Study  Process.  Because  the  objective  of 
our  study  was  to  evaluate  the  “TO-BE”  process,  we  did  not  seek  to  answer  any  particular 
fatigue  related  questions  with  this  study.  Instead,  using  our  newly  designed  templates, 
we  wanted  to  determine  if  data  could  be  collected,  manipulated  and  stored  more  easily 
and  with  less  data  corruption. 


Figure  38.  Naval  Officer  Indoctrination  School 


We  selected  the  Newport  location  primarily  because  the  daily  routine  of  OIS 
students  is  highly  regimented.  This  regimentation  allowed  us  to  more  easily  monitor  the 
data  collection  process  and  to  address  issues  as  they  arose.  Additionally,  the  staff  and 
faculty  at  OIS  understood  our  intent  and  graciously  allowed  us  to  evaluate  our  process  at 
their  facility.  Although  the  reengineering  process  was  the  focus  of  our  study,  we  did 
collect  human  fatigue  and  performance  data  on  students  and  faculty  at  OIS.  These 
actigraphy  and  ARES  performance  data  were  collected  on  20  participants  from  3 1  March 
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2003  to  5  April  2003.  We  conducted  only  preliminary  data  analysis  to  evaluate  the 
changes  we  had  incorporated  into  the  “TO-BE”  process,  however,  we  did  not  conduct  an 
in  depth  analysis  of  the  data.  Since  the  preliminary  analysis  revealed  interesting 
observations  with  regard  to  human  fatigue  and  performance,  we  have  made  our  OIS 
Study  data  available  for  further  analysis  by  fatigue  researchers  on  the  HSIL  web  server 
and  it  is  anticipated  that  NFS  students  will  use  the  data  for  further  analyses. 

Broken  into  phases,  this  chapter  provides  a  summary  of  the  OIS  Study  followed 
by  lessons  learned.  Using  the  templates  created  in  the  reengineering  effort,  we  tailored 
them  for  this  specific  study.  We  collected  survey  data  using  the  web  surveys,  actigraphy 
using  the  Actigraph  watches,  logging  data  using  the  event  logger  on  the  PDA  and 
performance  data  using  the  ARES  software,  also  on  the  PDA.  By  conducting  an  actual 
field  study,  we  were  able  to  experience  firsthand  the  pressures  facing  a  research  team  in  a 
field  environment.  Through  our  own  experience  and  observations,  we  were  then  able  to 
modify  the  templates  we  had  designed  in  order  to  provide  maximum  functionality  to 
future  research  teams.  The  results  of  this  OIS  study  serve  as  a  validation  of  the 
reengineered  efforts  reflected  in  the  “TO-BE”  process. 

B.  PHASE  1:  COLLECT  DATA  SUB -PROCESS 

In  the  weeks  preceding  the  OIS  study,  we  completed  tasks  similar  to  those 
required  by  fatigue  research  students  under  the  sub -process  activities:  Train  Research 
Team  and  Prepare  for  Data  Collection.  Using  the  “TO-BE”  process  additions,  we  revised 
the  Study  Repository  Template  database  on  the  HSIE  web  server 
(http  ://www  .nps  .navy  .mil/HumanEactors)  and  saved  the  newly  revised  OIS  Study 
Repository  database  to  the  server.  We  entered  the  research  team  information  and  the 
study  information  using  the  input  forms  in  the  database.  Once  the  decisions  had  been 
finalized  on  what  data  we  would  collect,  we  were  able  to  modify  the  Institutional  Review 
Board  (IRB)  proposal  documentation  template  and  submit  the  proposal  for  approval.  We 
saved  a  copy  of  the  IRB  proposal  documentation  to  the  server. 

After  the  IRB  proposal  was  approved,  we  modified  the  survey  and  log  templates 
to  fit  the  OIS  Study.  The  Demographic  questionnaire  and  Eark  and  Owl  surveys  were 
posted  to  the  website  and  PARTICIPANT  and  EARK  AND  OWL  tables  in  the  OIS  Study 

Repository  database  were  set  to  be  updated  as  data  was  entered  into  the  web  surveys. 
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The  use  of  append  table  queries  allowed  the  OIS  Study  database  to  automatically  check 
for  new  updates  on  the  web  server  and  append  them  upon  opening  of  the  database.  The 
PDA  Event  Log  Template  was  modified  to  allow  participants  to  log  when  they  took  the 
Actigraph  watch  off  and  put  it  back  on,  when  they  went  down  for  sleep  and  when  they 
awoke,  and  when  they  stood  watch.  The  newly  modified  log  was  called  Sleep_Log_OIS. 

Under  Prepare  Participant  Activity,  coordination  was  made  with  the  OIS  cadre  to 
identify  20  participants  for  the  study  and  to  ask  them  to  complete  the  web  surveys.  As 
they  completed  these,  we  could  monitor  their  submitted  survey  data  using  the  OIS  Study 
Repository  database.  If  any  survey  data  field  was  incomplete,  we  were  able  to  coordinate 
through  the  cadre  to  rectify  the  problem.  As  a  result,  we  were  able  to  collect  all  20  of  the 
participants’  survey  data  with  minimal  missing  values.  Participants  completed  online 
documentation  the  week  prior  to  the  study  start  date.  Had  we  been  actual  fatigue 
researchers,  the  Lark  and  Owl  data  could  also  have  been  evaluated  allowing  the 
opportunity  to  connect  momingnes  s/eveningness  information  with  each  participant.  As 
participants  were  added,  the  database  automatically  assigned  participant  ID  numbers  to 
each  person.  Using  these  numbers,  we  were  able  to  conduct  the  tasks  under  the  Prepare 
Equipment  Activity.  We  initialized  the  Actigraph  watches  and  assigned  each  to  a 
participant  using  the  database  input  form.  We  also  prepared  the  ARES  and  logging  PDA 
software  for  download  to  the  PDAs.  Because  we  were  going  to  use  the  PDAs  that  had 
been  issued  to  the  participants  by  OIS,  we  could  not  prepare  the  PDAs  until  we  arrived 
on  site. 

Arriving  a  day  before  the  actual  Actigraph  data  collection  was  scheduled  to  begin, 

we  were  able  to  meet  with  participants  to  collect  their  PDAs  and  coordinate  for 

redistribution  of  the  devices.  We  also  were  able  to  clarify  any  missing  entries  made  to 

their  survey  data  which  resulted  in  “already-cleaned”  Demographic  questionnaire  and 

Lark  and  Owl  data.  We  synchronized  each  participant’s  PDA  with  toe  standard  Palm 

Desktop  software  that  accompanies  the  PDA  software.  Since  these  PDAs  were  already  in 

use  by  the  participants,  we  could  not  rename  them  to  conform  to  our  naming  convention 

for  PDAs  of  HSILl,  HSIL2,  etc.  which  we  had  used  in  our  template  design.  This  would 

have  required  us  to  delete  everything  on  their  PDAs.  As  each  PDA  “user”  name  was 

added  to  our  synchronizing  software,  we  recorded  this  name  into  the  PDA  table  of  our 
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OIS  Repository  and  added  the  corresponding  Participant  ID  for  each  PDA.  Once  the 
PDAs  were  connected  with  Participant  IDs,  we  loaded  the  ARES  and  logging  software  to 
each  PDA.  We  also  loaded  the  newly  modified  OIS  Sleep  Log  file, 
Sleep_Log_OIS.PDB,  and  the  OIS  Sleep  Log  Input  Lorm  file, 
Sleep_Log_OIS_HDL.PDB,  to  each  PDA.  In  preparing  each  log,  we  selected  the 
appropriate  participant  ID  number  for  each  PDA  using  the  HanDBase  3.0  desktop 
software  and  these  two  files.  Although  this  participant’s  “ID#”  field  displays  on  the  PDA 
log  view  (Ligure  39),  the  user  cannot  modify  it. 


Figure  39.  PDA  Quick  Link  Buttons 


When  the  data  are  collected,  this  participant  ID  number  allows  the  researcher  to 
identify  whose  data  was  downloaded.  Similarly,  using  the  registration  menus  on  the 
ARES  software  on  the  PDA  itself,  we  registered  each  participant  using  the  corresponding 
participant  ID  and  we  set  each  to  use  the  Commander  Battery  of  tests.  This  battery 
includes  a  sleepiness  scale,  a  reaction  time  test  and  a  running  memory  test.  Because  the 
participants  were  in  a  classroom  environment,  we  chose  not  to  set  their  PDA  alarms  to 
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alert  them  to  take  their  ARES  tests  though  this  technique  proved  to  be  helpful  when  we 
tested  it  at  the  HSIL.  To  assist  participants,  we  did  set  the  Memo  Pad  and  To  Do  List 
buttons  on  the  bottom  of  the  PDA  (Figure  39)  to  open  the  Log  and  ARES  respectively  for 
quick  and  easy  data  entry. 

The  following  morning,  we  distributed  the  Actigraph  watches,  PDAs  and 
informed  consent  forms  and  instructed  the  participants  on  to  the  use  of  the  PDA  software. 
Once  the  signed  consent  forms  were  collected,  we  began  the  5-day  data  collection  phase. 
During  the  course  of  data  collection,  we  periodically  verified  data  integrity  by  collecting 
the  Actigraphs  and  PDAs  and  downloading  the  data.  One  Actigraph  device  failed  during 
the  study  and  a  replacement  watch  was  given  to  the  participant.  Throughout  this  process, 
the  participant  continued  to  log  his  activities. 

Once  the  data  collection  period  had  ended,  all  Actigraphs  and  PDAs  were 
collected.  We  synchronized  the  PDAs  using  the  Palm  Desktop  software  to  collect  the  log 
and  ARES  data,  deleted  the  ARES  and  logging  software  from  the  PDAs  and  then 
returned  the  PDAs  to  the  participants.  As  we  did  this,  we  used  the  Collected  Data  and 
Equipment  Form  on  the  OIS  Study  Repository  to  track  our  progress. 

At  the  end  of  the  Prepare  for  and  Collect  Data  Sub-Process,  we  had  PDA  data 
files  and  Actigraph  data  files  stored  on  the  computer  for  each  participant,  signed  consent 
forms  and  the  Actigraph  watches  with  stored  data.  Additionally,  the  OIS  Study 
Repository  database  included  completed  and  cleaned  STUDY,  RESEARCH  TEAM, 
PARTICIPANT,  and  LARK  AND  OWL  tables  and  cross-references  for  PDAs  and 
Actigraphs  to  the  participants  in  the  PDA  and  ACTIGRAPH  tables.  Formatted  reports 
were  also  available  from  the  OIS  Study  Repository  to  assist  in  the  data  analysis  phase. 

C.  PHASE  2:  PREPARE  DATA  FOR  ANALYSIS  SUB  -PROCESS 

We  began  this  phase  with  the  Download  PDA  Data  Activity.  As  a  result  of  the 
PDA  HotSync,  each  participant’s  sleep  log  data  included  in  the  file  Sleep_Log_OIS.PDB 
from  the  PDA  was  copied  to  the  C^Palm\[PDA_name]\Backup  location  on  the  laptop 
hard  drive.  From  the  HanDBase  desktop  software,  we  opened  each  of  these  files  and 
appended  each  participant  to  one  consolidated  Sleep_Log_OIS.PDB  file.  We  then 
exported  that  data  file  to  MS  Excel  and  copied  it  directly  to  the  LOG  table  in  the 
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repository  database.  The  pre- formatted  Log  Report  in  the  database  generated  a 
consolidated,  color  coded  log  of  each  of  the  participant’s  activities.  Similarly,  we 
prepared  the  ARES  data  by  loading  the  ARSubjDB.PDB  file  of  each  participant  stored  in 
the  C:\Palm\[PDA_name]\Backup  location  on  the  laptop.  Unlike  HanDBase  however, 
the  Data  Manager  software  does  not  have  an  append  function.  Therefore,  we  copied  each 
participant’s  ARES  data  directly  from  Data  Manager  to  the  ARES  DATA  table  in  the 
OIS  Study  Repository  database  using  the  cut  and  paste  function.  We  saved  all  of  the  log 
and  ARES  original  .PDB  files  to  the  HSIE  web  server  in  the  OIS  Study  Repository 
folder. 

Using  the  Actigraph  interface  device,  we  downloaded  the  actigraphy  data  using 
Act  Millennium  software  and  saved  each  file  with  the  serial  number  as  the  filename  as  a 
.DAT  file  and  .AMI  file  (e.g.,  SN01.DAT  and  SNOl.AMI).  Because  the  Actigraph  data 
files  can  only  be  opened  from  inside  the  Act  Millennium  and  Action  W  software,  there 
was  no  reasonable  way  to  store  this  data  into  the  database.  Therefore,  we  stored  both  of 
these  files  on  the  web  server.  The  ACTIGRAPH  table  in  the  database  provides  the  key  to 
identifying  the  corresponding  Actigraph  serial  number(s)  for  each  of  the  participants. 

Actigraphy  data  were  trimmed  and  marked  according  to  the  Eog  Report  of  the 

participants’  activity  using  Action  W  software.  The  data  files  were  saved  as  .AMI  files 

and  .EBE  files  with  an  “A”  added  to  the  filename  (e.g.,  SNOIA.AMI  and  SNOIA.EBE)  to 

allow  both  the  raw  and  trimmed  data  to  be  easily  identified.  All  of  these  files  were  stored 

on  the  web  server.  Using  the  summary  statistics  of  Action  W  for  each  participant,  the 

average  number  of  hours  of  sleep  per  night  calculated  for  the  participants  was  recorded. 

This  average  of  six  hours  was  used  to  precondition  the  preceding  three  days  of  the 

template  schedule  using  the  EAST  software.  Each  .EBE  file  was  then  loaded  into  the 

template  EAST  schedule  with  the  preconditioned  preceding  three  days.  Each 

participant’s  EAST  data  was  saved  as  a  EAST  schedule  using  the  .EAS  file  extension 

(e.g.,  SNOl.EAS).  These  EAST  schedules  were  also  added  to  the  OIS  Study  Repository 

folder  on  the  web  server.  Eor  the  participant  whose  watch  had  failed  and  had  been  issued 

a  second  watch,  we  were  unable  to  combine  the  data  from  both  watches  into  one  Action 

W  data  file;  both  sets  of  data  were  cleaned  and  stored  into  two  respective  files  based  on 

the  log.  Because  the  participant  kept  an  accurate  log  including  the  time  between  the  first 

74 


watch  failing  and  the  second  watch  beginning  to  record  data,  it  was  possible  to  manually 
input  the  sleep/wake  from  the  first  watch’s  failing  through  the  end  of  the  study  period 
into  FAST  using  the  Action  W  file  from  the  second  watch  and  the  log.  This  provided  an 
accurate  measure  of  the  participant’s  predicted  effectiveness  and  made  it  possible  for 
keep  this  participant’s  data  in  the  study.  After  all  Actigraph  data  were  “cleaned”  and 
predicted  cognitive  effectiveness  was  modeled  in  FAST  for  each  participant,  screen 
captures  of  the  trimmed  Action  W  files  and  the  FAST  schedules  for  each  participant  were 
also  imported  into  a  PowerPoint  presentation  and  saved  on  the  server. 

By  the  end  of  the  Prepare  Data  for  Analysis  Sub-Process,  all  participants’  raw  and 
“cleaned”  data  files  were  well- organized  on  the  web  server  for  easy  access  as  was  a 
summary  presentation  file  including  the  actigraphy  and  FAST  performance  effectiveness 
for  each  participant.  The  OIS  Study  Repository  database  included  updated  LOG  and 
ARES  tables  and  the  data  from  all  20  of  the  participants  was  prepared  for  analysis.  As 
projected  by  our  KVA  calculations,  we  expected  the  cycle  time  for  this  sub-process  to 
decrease  by  half.  Because  of  the  standardized  templates  and  storage  methods  as  well  as 
the  efficiently  coded  log  data  easing  the  burden  of  “cleaning”  actigraphy,  the  actual  cycle 
time  for  this  sub -process  for  this  study  was  less  than  one  forth  of  the  corresponding  sub- 
process  cycle  time  in  the  “AS-IS”  model. 

D.  PHASE  3:  ANALYZE  AND  PUBLISH  SUB -PROCESS 

Because  the  data  analysis  portion  of  this  sub-process  did  not  change  from  the 
“AS-IS”  process  to  the  “TO-BE”  process,  we  did  not  conduct  an  in-depth  analysis  of  the 
sleep  data.  We  did,  however,  request  the  assistance  of  research  analysts  to  verify  that  the 
data  could  be  analyzed  efficiently  in  their  stored  formats.  One  of  the  researchers  that 
assisted  us  in  this  evaluation  was  ET  John  Nguyen  who  had  conducted  the  USS  JOHN  C. 
STENNIS  Study.  In  his  assessment,  the  prepared  data  were  ready  for  analysis.  As  a 
result  of  the  standardized  formatting  and  efficient  activity  logs,  he  estimated  that  he  could 
have  saved  months  of  time  had  he  had  this  “TO-BE”  process  available  to  him  prior  to 
conducting  his  study.  A  second  researcher  looked  at  the  ARES  data  and  determined  that 
from  its  stored  format  in  the  repository  database,  it  could  have  easily  been  evaluated  to 
determine  possible  correlations  between  performance  and  fatigue.  There  were  some 
participants  that  did  not  complete  the  battery  of  tests  the  recommended  three  times  per 
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day.  Because  we  were  more  interested  in  whether  the  data  could  be  collected  and 
downloaded  efficiently,  we  did  not  adequately  emphasize  the  importance  of  completing 
all  required  testing.  In  the  future,  researchers  can  download  ARES  data  daily  to  ensure 
participant  compliance  for  complete  analysis. 

Due  to  the  ease  of  “cleaning”  the  Actigraph  data,  we  were  able  provide  the 
command  at  OIS  some  preliminary  feedback  on  the  participants’  sleep/wake  activity  as 
well  as  performance  effectiveness  during  the  collection  period.  This  was  the  first  time 
such  immediate  feedback  could  be  provided  to  a  sponsoring  organization.  As  a  result  of 
this  feedback,  the  command  requested  the  Actigraph  data  be  collected  for  an  additional 
period  to  evaluate  the  participants’  performance  during  a  battle  drill  exercise.  Though  no 
formal  data  analysis  feedback  was  requested,  the  command  was  able  to  see  the  potential 
value  provided  by  this  fatigue  study  process  to  maximize  student  performance  given  their 
limited  time  in  training.  Another  study  may  be  warranted  to  answer  specific  questions 
for  the  OIS  command  in  regard  to  fatigue  management.  Data  collected  from  this  study 
could  be  used  as  a  baseline  for  comparison  since  it  is  available  in  the  repository. 

E.  LESSONS  LEARNED 

As  a  result  of  this  validation  study,  we  were  able  to  experience,  firsthand,  the 
challenges  of  conducting  research  on  human  in  the  field.  In  this  context,  we  have  made 
some  valuable  observations  that  impact  future  studies  using  our  reengineered  process. 
The  following  paragraphs  summarize  these  observations  in  four  major  areas:  1) 
advantages  of  the  PDA  event  logger,  2)  PDA  issues,  3)  Actigraph  issues,  and  4)  ARES 
issues. 

1.  Advantages  of  Event  Logger 

Trimming  raw  Actigraph  data  is  time  consuming  using  Action  W  software.  To 
accurately  represent  a  participant’s  data,  researchers  compare  the  data  recorded  in  the 
sleep  and  activity  log  to  the  Actigraph  data  to  verify  whether  the  data  are  sleep  or  wake 
epochs  and  to  record  when  an  individual  was  down  for  sleep.  When  using  a  paper  log, 
the  researcher  has  to  either  transpose  the  log  data  to  a  spreadsheet  or  use  the  paper  itself 
to  compare  with  the  Actigraph.  In  both  methods,  extracting  the  useful  information  from 
the  log  is  tedious.  In  cases  such  as  the  USS  JOHN  C.  STENNIS  study,  the  inaccuracy  of 
the  paper  bgs  resulted  in  the  researcher  having  to  eliminate  the  use  of  logs  from  his 
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“cleaning”  process  only  after  many  hours  were  spent  trying  to  glean  useful  information 
from  them.  When  they  finally  were  eliminated,  the  researcher  had  to  use  a  technique  for 
trimming  sleep  data  based  on  assumptions.  Using  this  standard  method  for  trimming  data 
in  the  Action  W  software,  researchers  coded  epochs  where  activity  levels  fell  below  50 
per  epoch  for  20  or  more  minutes  as  sleep  periods.  If  the  activity  registered  as  a  0  for  10 
minutes  or  more,  the  researcher  assumed  the  participant  had  removed  the  Actigraph 
watch  and  coded  the  data  as  sleep  or  wake  periods  based  on  the  duration  and  time  of  day. 
All  other  activity  periods  were  classified  as  wake. 

Using  the  PDA  log,  participants  were  much  more  likely  to  log  their  daily  events 
because  the  method  was  easy  for  them  to  do  so.  When  comparing  log  data  with 
Actigraph  data,  the  Log  Report  made  “cleaning”  the  raw  Actigraph  data  very  simple. 
Figure  40  shows  a  screen  capture  of  the  method  used  during  this  “cleaning”  process.  The 
top  window  is  the  participant’s  raw  Actigraph  data  in  Action  W  software  and  the  bottom 
window  is  his  corresponding  log  data  from  the  Log  Report  from  the  OIS  Study 
Repository  database.  The  researcher  looks  for  the  highlighted  events  and  records  them 
using  the  Action  W  tools.  Using  the  participant  number  52’ s  data  from  the  OIS  study,  we 
were  able  to  “clean”  his  data  in  less  than  5  minutes  using  this  method.  Using  the  standard 
trimming  technique  that  the  USS  JOHN  C.  STENNIS  researcher  used  to  “clean”  the  data 
without  a  log  on  this  same  participant,  it  took  approximately  15  minutes.  These  results 
were  even  a  greater  improvement  than  we  had  estimated  in  our  KVA  analysis 
(APPENDIX  A).  We  tested  these  results  on  all  of  our  participant  data  and  found  that  on 
average,  the  standard  trimming  technique  took  more  than  three  times  as  long  to  “clean” 
Actigraph  data  than  when  using  the  PDA  log. 
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Figure  40.  “Cleaning”  Actigraph  Data  Using  Action  W  and  PDA  Log  Report 


Not  only  is  the  PDA  log  method  faster  than  the  standard  coding  technique  but  it 
also  appears  significantly  more  accurate  in  some  cases.  Because  of  the  convenience  of 
the  PDA  log,  participants  are  more  likely  to  maintain  their  logs.  Additionally,  because 
the  researchers  can  quickly  compare  log  data  with  Actigraph  data,  they  can  routinely 
conduct  data  downloads  to  assess  the  quality  of  data  collection  and  take  corrective 
measures  if  needed.  If  there  are  unexplained  data  points,  the  researchers  can  clarify  the 
issues  with  the  participants  while  still  there  and  while  events  are  still  fresh  in  their 
memory  allowing  the  logs  to  more  accurately  reflect  their  behavior. 

To  emphasize  the  impact  the  log  has  on  analyzing  a  participant’s  data,  we 
compared  the  standard  method  of  coding  used  in  prior  studies  to  the  method  of  using  the 
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PDA  log.  Using  participant  52  from  the  OIS  study  again,  Figure  41  shows  this  dataset 
trimmed  with  the  standard  coding  technique.  The  red  underlined  epochs  represent  sleep 
periods;  all  other  times  are  coded  as  wake.  The  aqua  filled  box  periods  represent  the 
times  a  participant  was  down  for  sleep.  Using  this  same  participant  in  a  comparative 
example,  the  Actigraph  data  were  trimmed  using  the  PDA  based  log  (Figure  42). 
Because  the  activity  levels  did  appear  unusually  high  for  sleep  epochs,  we  asked 
participant  52  to  verify  the  sleep  log  and  verify  our  assessment  of  his  actigraphy  while 
still  on  site.  He  confirmed  the  events  he  had  logged  and  assured  us  he  had  not  awoken 
during  his  nights  of  sleep.  Figure  43  shows  a  side  by  side  comparison  of  the  Actigraph 
data  trimmed  using  both  techniques.  Without  the  log,  many  epochs  were  coded  as  wake 
as  they  did  not  fall  below  the  threshold  of  50  for  at  least  20  minute  intervals.  In  the  data 
“cleaned”  using  the  standard  coding  technique,  it  appears  as  if  the  participant  is  getting 
much  less  sleep  than  he  actually  was. 


Tue  4/U2003 
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Figure  41.  Actigraph  Data  Trimmed  Using  Standard  Method 
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Trimmed  Using  Standard  Method  Trimmed  with  PDA  Log 

Figure  43.  Side  By  Side  Comparison  of  Actigraph  Data 


We  imported  the  data  from  both  methods  into  FAST  sehedules  to  model 
partieipant  52’ s  predicted  effectiveness.  Figure  44  shows  the  FAST  model  of  the 
Actigraph  trimmed  using  the  standard  method  and  Figure  45  shows  the  FAST  model 
using  the  PDA  log  to  trim  the  data.  The  red  line  indicates  the  sleep  reservoir  as  it  is 
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depleted  and  refilled  according  to  the  sleep/wake  epochs.  Figure  46  shows  a  side  by  side 
comparison  of  the  FAST  models.  The  modeled  effectiveness  of  the  two  is  decisively 
different.  The  additional  sleep  coded  using  the  PDA  log  and  the  testament  of  the 
participant  immediately  after  the  data  had  been  collected  to  verify  accuracy  accounts  for 
a  much  higher  predicted  effectiveness  level.  Without  any  log,  the  model  indicates 
predicted  effectiveness  far  below  acceptable  levels.  Though  it  appeared  from  his  raw 
Actigraph  data  that  participant  52  did  not  get  much  rest,  according  to  his  report,  he  did. 
This  report  made  by  the  participant  is  his  subjective  account.  However,  the  assumptions 
made  by  the  researcher  without  reference  to  a  log  are  also  subjective  and  arguably  less 
accurate.  Subsequent  examination  of  the  Actigraph  used  by  participant  52  showed  that 
the  device  needed  factory  calibration. 
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Figure  44.  FAST  Model  without  a  Log 
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Figure  45.  FAST  Model  with  a  PDA  Log 


Trimmed  Using  Standard  Method 


Trimmed  with  PDA  Log 


Figure  46.  Side  By  Side  Comparison  of  FAST  Models 


Because  our  focus  during  the  validation  study  was  on  collecting  and  storing  data 
efficiently,  we  did  not  concentrate  on  the  actual  fatigue  data  as  sleep  researchers  would 
have.  Had  we  been  interested  in  answering  fatigue  related  questions,  we  could  have 
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conducted  routine  data  downloads  of  Actigraph  and  log  data  and  conducted  quick 
analysis  to  ensure  participants  are  logging  events  effectively.  Using  the  old  method,  the 
time  it  took  researchers  to  verify  Actigraph  data  with  the  log  data  made  this  comparison 
impossible  to  do  while  on  site  without  interrupting  the  study.  The  addition  of  the  PDA 
log  allows  researchers  to  conduct  this  data  verification  between  Actigraph  data  and 
logged  events  very  easily.  Any  issues  noted  by  the  researeh  team  can  be  resolved 
immediately  with  the  participant  thus  increasing  the  accuracy  of  the  analysis. 

2.  PDA  Issues 

As  part  of  our  reengineering  research,  we  evaluated  various  PDA  handhelds  and 
chose  the  Handspring  TREO  for  our  reengineered  process.  The  laptops  we  used  were 
equipped  with  Universal  Serial  Bus  (USB)  ports.  When  comparing  ease  of  using  the 
HotSync  function,  the  Handspring  versus  the  Palm  products  we  evaluated  seemed  easier 
to  use.  Because  the  HanDBase  and  ARES  software  work  with  any  PDA  using  the  Palm 
OS,  however,  the  process  was  not  dependent  on  the  PDA  model.  When  we  chose  OIS  as 
our  validation  study  location,  we  understood  that  students  and  cadre  were  issued  Palm 
PDAs  upon  arrival.  Because  we  had  not  yet  bought  PDAs  for  HSIE,  we  decided  to  use 
the  participants’  Palms.  We  requested  from  OIS  a  sample  Palm  PDA  so  we  could  load 
and  test  our  software  to  ensure  compatibility.  A  Palm  ml 00  arrived  and  although  the 
display  was  black  and  white,  all  software  seemed  compatible.  The  Palm  mlOO  cradle 
used  a  serial  connector  to  the  laptop  for  HatSync  functions.  When  we  arrived  on  site,  we 
found  that  all  participants  were  using  the  Palm  V  version  of  PDAs.  When  we  tried  to 
perform  a  HotSync  function  with  our  laptop  using  the  Universal  Serial  Bus  (USB) 
connection  from  the  Palm  V  cradle  to  the  laptop  and  Palm  Desktop  software,  there  were 
errors  that  could  only  be  corrected  by  uninstalling  and  reinstalling  the  Palm  software. 
Even  after  this,  we  experienced  occasional  HotSync  errors  with  the  Palm  V  using  the 
USB  port.  Additionally,  we  had  to  spend  hours  configuring  the  Palm  V’s  with  the  data 
collection  software  the  evening  before  eollection  began.  To  avoid  corruption  of  the 
participants’  saved  information  on  their  PDAs,  we  chose  to  use  their  PDA  names  they 
had  established  and  deviate  from  our  process’  naming  convention  (e.g.,  HSIEl,  HSIE2, 
etc.).  This  caused  us  to  have  to  change  the  repository  database  PDA  STATUS  table  and 
populate  it  with  the  OIS  PDA  names.  Because  many  participants  used  their  last  name  as 
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their  PDA  name,  this  collected  data  became  sensitive  since  the  HSIL  policy  prohibits 
storing  any  operational  data  with  information  identifying  the  participants  other  than  their 
participant  ID  number.  In  order  to  store  the  data  on  the  HSIL  server,  we  had  to  delete 
these  PDA  names. 

We  recommend  for  future  studies  that  the  HSIL  purchase  30  of  the  Handspring 
TREO  models.  When  installing  each  PDA  on  the  HSIL  operational  study  laptop  we 
recommend  that  PDAs  be  named  according  to  our  naming  convention:  HSILl  through 
HSIL30.  Each  should  be  loaded  with  the  HanDBase  and  ARES  software.  Additionally, 
we  recommend  loading  the  Sleep  Eog  Template  and  its  accompanying  input  form  created 
with  HanDBase  software  onto  each  PDA.  Eor  both  the  log  and  the  ARES  programs,  the 
participant  ID  numbers  should  be  added.  These  ID  numbers  should  correspond  to  the 
PDA  number.  Eor  example,  on  the  PDA  named  HSIEl,  the  ARES  registered  user  should 
be  01  and  the  “ID#”  field  on  the  log  form  should  be  01.  This  way,  all  data  downloaded 
from  ARES  and  the  event  log  programs  during  the  HotSync  function  will  be 
appropriately  marked  with  a  participant  ID  number  that  corresponds  to  the  PDA  ID  in  the 
Study  Repository  database. 

In  the  event  that  participants  are  already  issued  PDAs  from  a  U.S.  governmental 
agency,  we  recommend  that  the  research  team  attempt  to  accommodate  the  participants 
and  use  their  issued  PDAs  so  as  not  to  further  inconvenience  the  participants  by  requiring 
them  to  carry  a  second  PDA  during  the  data  collection  phase.  In  this  case,  it  is  essential 
that  the  research  feam  acquire  a  sample  of  the  PDA  model  used  by  the  participants  to 
ensure  compatibility  with  both  software  and  hardware.  Additionally,  the  research  team 
should  allow  enough  time  after  arrival  on  site  and  prior  to  data  collection  to  configure  the 
participants’  PDAs  accordingly.  Einally,  the  research  team  must  ensure  that  the  data 
saved  from  each  PDA  after  each  HotSync  under  the  Palm  directory  on  the  laptop  is 
erased  and  any  identifying  PDA  ID  names  in  the  Study  Repository  database  are  erased  as 
well  to  protect  the  participants’  identity. 
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Despite  the  technical  obstacles,  the  use  of  PDAs  undoubtedly  improves  the  “AS- 
IS”  process.  Whether  using  the  PDAs  from  the  HSIL  or  participants’  PDAs,  the  ability  to 
collect  event  log  data  and  ARES  data  provides  a  tremendous  advantage  to  the  research 
team. 

3.  Actigraph  Issues 

Collecting  actigraphy  using  the  Actigraph  and  associated  software  posed  unique 
challenges.  Issues  involving  the  Actigraph  watches  themselves  as  well  as  observations  of 
other  hardware  and  software  became  apparent  to  us  during  the  study. 

The  Actigraph  watch  itself  was  bulky  for  the  participants  as  they  noted  on  their 
exit  surveys;  some  cadre,  unaware  of  the  study,  required  the  student  participants  to 
remove  their  Actigraphs  for  inspections.  This  resulted  in  a  few  participants  failing  to  put 
the  watch  back  on  for  several  hours.  A  smaller  more  watch- like  Actigraph  with  long 
battery  life  and  low  failure  rates  would  clearly  be  better  suited  for  field  studies. 

The  interface  box  for  the  Actigraph  requires  a  serial  port  connection  and  is  not 
available  for  use  with  a  USB  connector.  Because  many  laptops  are  no  longer  being  made 
with  serial  ports,  this  may  become  an  issue  in  the  future.  In  a  similar  incompatibility 
issue,  the  software  accompanying  the  watches  from  the  manufacturer  is  a  DOS  based 
program  that  will  not  run  on  any  Windows  Operating  System  higher  than  Windows  95. 
The  Act  Millennium  software  offered  by  AMI,  the  marketing  company  of  the  Actigraph 
watches,  as  a  Windows  alternative  is  compatible  with  Windows  XP.  However,  the 
initialization  of  and  downloading  of  data  from  the  watches  are  not  intuitive  processes. 
Researchers  must  be  very  familiar  with  the  configuration  settings  of  this  software  prior  to 
collecting  data. 

Once  the  raw  data  have  been  downloaded  using  the  interface  unit  and  Act 
Millennium,  they  are  saved  as  a  .DAT  file  and  then  saved  as  an  .AMI  file  to  allow  Action 
W  software  to  read  the  data.  Once  trimmed  in  Action  W,  the  data  file  has  to  be  saved  as 
an  .EBE  file  to  be  read  with  EAST.  After  the  research  team  has  built  a  schedule  in  EAST 
covering  the  collection  period,  the  .EBE  file  is  imported  into  the  schedule.  After  it  has 
been  “cleaned”  in  EAST,  the  data  are  again  saved  with  a  .EAS  extension  so  it  can  be 
opened  as  a  schedule  in  the  future.  Because  of  the  unique  data  formats  for  these 
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programs,  the  data  could  not  be  saved  in  the  database  for  export.  Instead,  all  versions  of 
data  with  unique  file  extensions  were  saved  in  the  same  folder  as  the  database  on  the  web 
server.  We  tried  to  save  the  hyperlink  to  this  folder  location  in  the  database  but  the  files 
could  not  be  associated  with  the  programs  to  open  automatically.  Once  a  file  is  saved 
from  Act  Millennium  as  an  .AMI  file  for  Action  W  for  example,  we  could  not  double 
click  on  that  file  and  have  Action  W  open  and  load  the  data.  This  process  of  opening 
programs,  manipulating  data,  saving  in  new  file  formats  to  new  locations  on  the  server 
proved  to  be  tedious  and  cumbersome.  This  inefficiency  caused  delays  in  preparing  data 
for  analysis  that  could  easily  be  overcome  by  allowing  import  and  export  of  universal 
data  formats  in  all  of  the  programs. 

To  ensure  data  integrity,  researchers  must  be  familiar  with  Act  Millennium, 
Action  W,  and  FAST  software  programs  well  before  the  data  collection  phase  begins. 
Even  with  a  comprehensive  understanding,  the  process  for  tracking  raw  and  “cleaned” 
data  files  quickly  becomes  overwhelming  to  the  inexperienced  researcher. 

4.  ARES  Issues 

Use  of  the  ARES  test  battery  requires  researchers  to  plan  for  a  significant  increase 
in  PDA  power  usage,  unforeseen  PDA  system  failures  and  unique  data  storage 
requirements.  Despite  some  difficulties  encountered  using  ARES  software,  the  data  from 
the  performance  tests  were  collected  and  the  use  of  ARES  is  now  a  viabb  metric 
available  for  future  studies. 

Taking  the  ARES  tests  battery  drains  the  PDA  power  rather  quickly.  As  a  result, 
researchers  must  be  sure  to  remind  participants  to  charge  their  PDAs  daily.  If 
participants’  PDAs  require  disposable  batteries,  it  is  recommended  that  the  research  team 
maintain  an  adequate  supply  of  these  batteries  on  hand  throughout  the  entire  data 
collection  period. 

As  a  result  of  a  bug  in  the  ARES  software,  two  PDAs  locked  up  during  the  OIS 
study.  The  data  in  one  device  was  unaffected  because  the  participant  brought  the  PDA  to 
us  and  we  initiated  a  soft  reset.  However,  the  data  in  the  second  device  was  lost  because 
the  participant  inadvertently  initiated  a  hard  reset.  Eortunately,  it  was  on  the  first  day  of 
collection  so  only  data  one  set  of  ARES  data  was  lost.  Eor  future  studies,  researchers 
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must  obtain  the  latest  version  of  ARES  software  to  minimize  the  risk  of  bugs  and  ensure 
participants  are  aware  of  hard  and  soft  reset  procedures  for  PDAs  in  the  event  there  is  a 
problem.  It  is  also  recommended  that  short  procedure  cards  be  inserted  into  the  PDA 
cases  provided  for  the  study.  This  card  can  contain  contact  information  for  researchers, 
soft  reset  procedures,  and  brief  “how-to”  instructions  for  completing  ARES  tests  and 
logging  events. 

When  downloading  ARES  data,  there  are  two  means  available  to  the  researcher. 
Eirst,  the  desktop  Data  Manager  program  can  be  opened  while  the  PDA  is  connected  to 
the  computer  and  the  database  file  can  be  directly  loaded  into  Data  Manager.  The  second 
method  is  by  using  the  HotSync  function.  Because  the  log  data  also  have  to  be 
downloaded  from  the  PDA,  the  HotSync  method  is  more  convenient.  Once  the  file  is 
downloaded  as  C\Palm\[PDA_Name]\ARSubjDB.PDB,  the  Data  Manager  program  has 
to  be  opened  and  only  from  there  can  the  researcher  load  this  data  file.  The  Data 
Manager  Program  will  not  accept  any  other  .PDB  file  than  one  named  ARSubjDB.PDB. 
Therefore,  all  of  the  ARES  raw  data  files  from  the  participants  cannot  be  stored  in  the 
same  folder.  Additionally,  because  the  files  all  have  the  same  name,  there  is 
understandable  confusion  manipulating  these  files  and  copying  and  storing  the  data  to  the 
database.  Additionally,  once  the  raw  data  are  consolidated  in  the  ARES  table  in  the 
database,  it  can  no  longer  be  loaded  into  Data  Manager  but  only  used  in  programs  like 
MS  Excel.  This  significantly  hinders  the  research  team  by  requiring  them  to  track  many 
storage  locations  for  the  same  data. 

Though  our  data  has  yet  to  be  analyzed  thoroughly,  we  are  confident  that 
collecting  ARES  data  on  the  PDA  is  an  option  now  available  to  HSIE  researchers. 
Diligent  monitoring  of  and  acquiring  the  new  releases  of  upgraded  versions  of  ARES 
software  is  imperative  as  some  of  these  obstacles  may  be  resolved  in  later  software 
versions. 

F.  CONCLUSION 

The  OIS  Study  served  to  validate  the  improvements  made  in  the  “TO-BE” 

process.  The  data  collection  techniques  do  streamline  the  process,  making  the  data 

analysis  more  efficient  and  accurate.  While  some  recommendations  for  particular 

hardware  and  software  used  must  be  considered,  the  results  of  this  study  show  that 
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progression  from  the  “AS-IS”  to  the  “TO-BE”  process  is  not  only  feasible  but  necessary. 
Using  the  reengineered  system  will  allow  more  studies  to  be  conducted  annually  with 
more  valuable  analysis  due  to  better  accuracy.  Additionally,  the  storage  repository 
allows  a  significant  capability  of  conducting  meta- analyses  whose  value  has  yet  to  be 
realized.  The  importance  of  fatigue  research  to  the  military,  in  particular,  demands  this 
immediate  change  from  the  current  process  to  the  reengineered  one. 
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m  CONCLUSIONS 


A.  SUMMARY  DISCUSSION 

The  tremendous  challenges  faced  by  scientists  conducting  human  research  are 
further  magnified  when  research  is  moved  from  the  controlled  laboratory  to  the  field 
environment  required  by  operational  research.  In  the  particular  field  of  human 
performance  and  fatigue  research,  the  unique  intricacies  of  data  collection  represent  some 
of  the  most  complex  in  the  field  of  human  research.  Serving  as  a  Proof  of  Concept  for  a 
representative  human  research  process,  the  process  of  conducting  human  performance 
and  fatigue  research  at  the  HSIL  at  NPS  was  used  to  substantiate  effective  data  collection 
and  storage  techniques.  Applying  current  information  technologies,  we  were  able  to 
demonstrate  a  significant  improvement  in  productivity  from  the  current  “AS -IS”  process 
to  the  reengineered  “TO-BE”  process.  The  methods  used  in  this  Proof  of  Concept  could 
be  applied  to  many  data  collection  and  storage  processes  in  the  broader  field  of  human 
research  thus  refining  the  effectiveness  of  operational  studies  as  researchers  as  they  strive 
to  improve  human  performance. 

Using  a  three-tiered  approach  in  the  redesign,  various  graphical  user  interface 
means  were  utilized  to  collect  and  store  data  in  a  relational  database  housed  on  a  NPS 
web  server.  Replacing  paper  surveys,  web  surveys  and  PDA-based  input  forms  were 
incorporated  to  collect  participant  data.  Similarly,  database  forms  were  designed  to  allow 
researchers  to  record  details  of  a  particular  study.  A  PDA-based  version  of  human 
performance  software  was  used  to  collect  performance  data  while  wrist  activity  monitors 
containing  accelerometers  recorded  participant  activity  levels.  The  database  served  not 
only  as  a  storage  facility  to  organize  and  maintain  the  collected  data  but  also  provided  the 
research  team  with  a  standardized  method  to  track  their  entire  research  process.  As  a 
result  of  the  insertion  of  various  information  technologies,  the  reengineered  process 
included  automating  current  sub -processes  and  tasks  and  more  importantly  enabling  new 
processes  to  be  incorporated.  With  the  conservatively  estimated  ROK  improvement  from 
12%  realized  by  the  current  process  to  26%  offered  by  the  reengineered  process, 
researchers  at  the  HSIL  at  NPS  will  be  able  to  conduct  more  studies  per  year  while  each 
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study  produces  more  accurate  data  analysis  results.  Additionally,  the  standardized 
storage  means  provided  in  the  “TO-BE”  process  enables  researchers  to  conduct  meta¬ 
analysis;  a  capability  not  possible  with  the  “AS-IS”  process. 

B.  RECOMMENDATIONS 

Because  reengineering  is,  in  itself,  an  iterative  process,  the  prototype  system 
developed  for  this  Proof  of  Concept  was  the  first  step.  Near-term  initiatives  taken  at  the 
HSIL  at  NPS  to  implement  this  redesign  will  result  in  greater  overall  improvement  to  the 
process.  Efforts  made  to  increase  compatibility  between  systems  by  human  performance 
and  fatigue  research  hardware  and  software  developers  will  further  improve  not  only  the 
process  at  NPS  but  the  more  extensive  field  of  human  performance  and  fatigue  research. 
Einally,  to  transform  the  impact  human  research  has  on  improving  human  performance 
under  constraining  conditions,  system  developers  can  take  steps  to  design  interoperable 
and  unobtrusive  collection  means  that  allow  for  immediate  analysis  of  human  systems 
thus  providing  decision  makers  with  the  feedback  required  to  optimize  human  resources. 

1.  Near-Term  Initiatives  to  Improve  the  Process  at  the  HSIL  at  NPS 

In  order  to  capitalize  on  the  reengineering  effort,  it  is  necessary  for  the  HSIE  to 
employ  an  IT  professional  to  implement  the  prototype  system  described  in  the  “TO-BE” 
process  and  provide  adequate  system  maintenance.  This  thesis  serves  as  a  guide  for  the 
research  team,  including  the  additional  IT  professional,  at  the  HSIE  to  package  a  “How- 
To”  type  toolkit  for  future  researchers  to  utilize  when  conducting  field  studies.  The  range 
of  techniques  for  data  collection  extend  beyond  the  methods  used  in  this  Proof  of 
Concept  system  and  each  should  be  modeled  to  provide  researchers  an  inventory  of  types 
of  data  to  collect  and  the  instructions  and  means  to  collect  each  in  various  operational 
environments.  Secondly,  templates  should  be  created  for  each  additional  data  collection 
method  available  to  the  HSIE  researchers  but  not  used  in  this  Proof  of  Concept.  Einally, 
a  web-based  feedback  form  should  be  created  to  capture  the  techniques  used  by  research 
teams  including  both  successful  and  unsuccessful  trials.  This  cumulative  feedback 
repository  will  include  the  data  and  analysis  of  all  studies  conducted  at  NPS  and  should 
capture  all  lessons  learned  by  each  research  team  thus  creating  a  venue  for  continual 
improvement  of  human  performance  and  fatigue  research  processes  at  NPS.  This  transfer 
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of  corporate  knowledge  to  IT  resources  will  provide  future  research  teams  an  invaluable 
source  of  information  allowing  them  to  maximize  the  potential  effects  of  their  research 
on  human  performance. 

2.  Immediate  Improvements  to  Human  Performance  and  Fatigue 
Research  Hardware  and  Software 

Though  current  hardware  and  software  developments  in  the  field  of  human 
performance  and  fatigue  research  have  only  recently  made  operational  studies  possible, 
straightforward  improvements  to  the  existing  hardware  and  software  could  have  a 
tremendous  impact  on  the  productivity  of  the  human  performance  and  fatigue  research. 
Storing  data  from  various  data  collection  media  into  compatible  formats  will  drastically 
reduce  the  time  and  effort  required  of  researchers  to  prepare  their  data  for  analysis. 
Further,  an  interface  application  to  provide  one  point  from  which  the  researcher  can 
access  all  software  programs  would  ease  the  burdens  associated  with  data  “cleaning”  thus 
allowing  more  time  to  spend  on  analyzing  the  collected  data.  This  application  should 
include  the  tools  required  to  download,  “clean,”  and  analyze  data  as  well  as  the  means  to 
efficiently  both  store  the  data  and  published  the  analysis.  A  more  far  reaching,  yet 
attainable,  goal  should  be  to  create  a  single  suite  of  hardware  and  software  that  allows 
researchers  to  choose  from  an  inventory  of  unobtrusive  data  collection  media  required  for 
their  research  and  easily  tailor  that  media  to  fit  the  operational  constraints  of  their  specific 
data  collection  effort.  This  data  collection  means  should  incorporate  wireless 
technological  advances  to  allow  data  transfer  without  human  intervention  thus  further 
increasing  accuracy  and  saving  time.  The  suite  should  also  provide  researchers  with  a 
variety  of  analysis  tools  to  enable  more  comprehensive  thus  more  revealing  analysis.  As 
new  hardware  monitoring  devices  are  designed  for  human  performance  and  fatigue 
research,  they  must  be  developed  with  system  interoperability  in  mind. 

3.  Long  Term  Designs  for  Human  Research  Product  Developers 

To  reach  beyond  data  collection  and  analysis  by  offering  solutions  to  mitigate 

obstacles,  developers  in  the  field  of  human  research  can  make  a  tremendous  impact 

toward  improving  the  human  condition.  With  the  continual  development  of  biomedical 

monitoring  devices,  developers  can  design  products  that  not  only  monitor  various  human 

systems  but  offer  self- analyzing  feedback  of  the  particular  system’s  performance.  From 

these  various  human  system  sensors  as  well  as  environmental  monitoring  media, 
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developers  will  be  able  to  create  systems  to  evaluate  the  feedback  and  provide 
recommendations  for  physical  or  behavioral  modifications  to  improve  human 
performance.  As  a  result  of  their  complexity,  humans  provide  invaluable  and  often 
irreplaceable  additions  to  many  processes.  Optimizing  human  performance  under 
constrained  conditions  is  the  goal  of  industry  leaders  and  military  commanders  alike. 
Focusing  human  research  product  development  on  this  goal,  researchers  will  have  the 
tools  required  te)  offer  leaders  options  for  resolving  conflicts  that  inhibit  optimal  human 
performance. 
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APPENDIX  A.  KVA  “AS -IS”  ROK 
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Process:  NPS  Executing  Fatigue  Studies 


Sub  Processes 

Weighted 

Learning 

Time 

Total 

Cost 

As-Is 

ROK 

To-Be 

ROK 

Dramatic 

ROK 

Collect  Data 

289 

3300 

9% 

1810 

4246 

43% 

395760 

370425 

107% 

Prepare  for  Analysis 

1158 

8278 

14% 

3075 

9185 

33% 

1204500 

2062500 

58% 

Analyze  and  Publish 

1161 

10192 

11% 

6885 

32082 

21% 

254250 

1122413 

23% 

TOTAL 

2608 

21770 

12% 

11770 

45513 

26% 

1854510 

3555338 

52% 
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APPENDIX  B.  WEB  SURVEY  PROTOTYPES 


Lark  and  Owl  Web  Survey 


Naval  Postgraduate  School  Bll^= 

Monterey.  C  a  i  •  c  r  n  a 


Lark  and  Owl 

Instructions: 

Take  the  following  survey  to  help  us  determine  if  you  are  a  lark,  and  owl  or  somewhere  in 
between.  These  questions  should  be  answered  as  if  you  were  at  home  and  not  deployed? 

Last  Name  I  Type  your  last  name 

Circadian  Identity: 

1.  During  your  last  vacation  week,  how  often  did  you  get  up  iater  than  pianned  or  have 
difficuity  getting  ready  on  time  even  though  you  went  to  bed  at  your  regular  time? 

^  It  never  happened 

^  It  happened  once 

^  It  happened  two  or  more  times 

^  It  happened  more  than  three  times 

2.  When  you  have  no  commitments  the  next  day,  at  what  time  do  you  go  to  bed  compared 
with  your  usual  time? 

^  Seldom  or  never  later 

^  Not  more  than  one  hour  later 

^  Between  one  and  two  hours  later 


Demographic  Web  Survey  Template 


Local  Demographic  Survey 
Instructions: 

Please  fill  out  all  blocks  in  this  survey,  when  finished  please  press 
the  "SUBMIT"  button 

1. NAME:  Last;  |  First:  |  MI:| 

2.  Rank:  fo^ 

3.  Service:  fuSN  3  Prior  Service  I”  Yes 

4.  E-mail  Address:  |N/A 

5.  Home  Phone:  | 

(SSS)S5S-S555 

6.  Pager  Number/Cell;  Fva 

(555)555-5555 

7.  Home  Address:! 

City;|  State:  I CA  “^1 

8.  Age:  j 

9.  Gender:  Male  ^  Female 

10.  Height:  M  ^IfeetlO  "clinches  (round  to  the  nearest  inch 

11.  Weight:!  lbs 

12.  Do  you  wake  up  to  an  alarm  clock?  yes  No 

13.  What  time  of  day  do  you  typically  leave  campus?  )1200-1^00  ^ 

14.  In  the  past  four  weeks,  how  much  have  you  been  able  to  workout?  (physical  activity) 

^  0-1  times  per  week  4-5  times  per  week 
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Morningness/Eveningness  Scale 


Directions:  This  scale  is  designed  to  help  you  identify  a  person’s  tendency  toward  a 
morning  (lark)  or  evening  (owl)  circadian  rhythm  pattern. 

1.  During  your  last  vacation  week,  how  often  did  you  get  up  later  than  planned  or 
have  difficulty  getting  ready  on  tie  even  though  you  went  to  bed  at  our  regular  time? 

A.  It  never  happened 

B.  It  happened  once 

C.  It  happened  two  or  more  times 

D.  It  happened  more  than  three  times 

2.  When  you  have  no  commitments  the  next  day,  at  what  time  do  you  go  to  bed 
compared  with  your  usual  time? 

A.  Seldom  or  never  later 

B.  Not  more  than  one  hour  later 

C.  Between  one  and  two  hours  later 

D.  More  than  two  hours  later 

3.  Suppose  that  you  have  decided  to  exercise  twice  a  week  with  a  friend.  The  only 
time  your  friend  can  make  it  is  from  7:00  a.m.  to  8:00  a.m.,  twice  a  week.  Assume  you 
decide  to  go  at  these  times.  Taking  into  account  how  you  usually  feel  in  the  morning, 
how  would  you  do? 


4. 

sleep? 


A.  Very  well 

B.  Well 

C.  Poorly 

D.  Very  Poorly 

At  what  time  in  the  evening  do  you  usually  start  feeling  tired  and  in  need  of 

A.  8:00  p.m.  -  9:30  p.m. 

B.  9:31  p.m.  -  10:45  p.m. 

C.  10:46  p.m.  -12:30  a.m. 

D.  12:31  a.m.  -1:45  a.m. 

E.  1:46  a.m.  -  3:00  a.m. 


5.  Suppose  that  you  were  able  to  choose  your  own  working  hours.  Assume  that  your 
job  was  interesting  and  paid  according  to  results.  Which  one  of  the  following  3- hour 
blocks  would  be  your  most  preferred  work  time? 


A. 

B. 


4:00  a.m.  -  7:00  a.m. 
7:00  a.m.  -  10:00  a.m. 
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C.  11:00  a.m.  -  2:00  p.m. 

D.  4:00  p.m. -7:00  p.m. 

E.  9:00  p.m.  -  12:00  Midnight. 

6.  One  sometimes  hears  about  “feeling  best  in  the  morning”  or  “feeling  best  in  the 
evening”  types  of  people.  Which  do  you  consider  yourself? 

A.  Definitely  a  “morning”  type 

B.  More  a  “morning”  than  an  “evening”  type 

C.  More  an  “evening”  than  a  “morning”  type 

D.  Definitely  an  “evening”  type 

SCORE  GUIDE 


Question 

Answer 

Score 

Question 

Answer 

Score 

No. 

No 

1 

A 

6 

4 

A 

7 

B 

5 

B 

6 

C 

5 

C 

5 

D 

4 

D 

4 

E 

3 

2 

A 

7 

5 

A 

7 

B 

6 

B 

6 

C 

4 

C 

5 

D 

3 

D 

4 

E 

3 

3 

A 

7 

6 

A 

9 

B 

6 

B 

7 

C 

4 

C 

3 

D 

3 

D 

1 

SCORE  SCALE 

17  to  19  Extreme  Owl 

20  to  23  Moderate  Owl 

24  to  29  Mild  Owl 

30  Neither  Owl  nor  Eark 

31  to  36  Mild  Eark 

37  to  40  Moderate  Eark 

41  to  43  Extreme  Eark 
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APPENDIX  C.  STUDY  REPOSITORY  TEMPLATE  DATABASE 

RELATIONSHIPS 


Actigraph  StatusActigraph 


Actigraph 

Status 

Actigraph 

Actigraph  ID  1 

?  actigraphID 

Attributes: 

Enforced 

RelationshipType: 

One-To-Many 

ParticipantActigraph 


Participant 

Actigraph 

ParticipantID  1 

?  participantID 

Attributes:  Enforced 

RelationshipType:  One-To-Many 


ParticipantARES  Data 


Participant 

ARES  Data 

ParticipantID  I 

?  participantID 

Attributes:  Enforced 

RelationshipType:  One-To-Many 


ParticipantCollected  Equipment  and  Data 


Participant 

Collected 

Data  and 

Equipment 

ParticipantID  I 

1  ParticipantID 

Attributes: 

Unique,  Enforced 

RelationshipType: 

One-To-One 
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ParticipantLark  and  Owl 


Participant 

Lark  and 
Owl 

ParticipantID  1 

?  ParticipantID 

Attributes: 

Enforced 

RelationshipType: 

One-To-Many 

ParticipantLog 


Participant 

Log 

ParticipantID  I 

?  ParticipantID 

Attributes:  Enforced 

RelationshipType:  One-To-Many 


ParticipantPDA 


Participant 

PDA 

ParticipantID  I 

?  participantID 

Attributes:  Enforced 

RelationshipType:  One-To-Many 


PDA  StatusPDA 


PDA  Status 

PDA 

PDA  ID 

I 

?  PDA  ID 

Attributes:  Enforced 

RelationshipType:  One-To-Many 
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Research  TeamResearch_Team_Study 


Research 

Research_Team_Study 

Team 

ResearcherlD  1 

?  ResearcherlD 

Attributes:  Enforced 

RelationshipType:  One-To-Many 


StudyParticipant 


Study 

Participant 

StudylD 

1 

?  Study  ID 

Attributes:  Enforced 

RelationshipType:  One-To-Many 

Study  Research_T  eam_Study 


Study 

Research_Team_Study 

StudylD 

1 

?  StudylD 

Attributes:  Enforced 

RelationshipType:  One-To-Many 
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APPENDIX  D.  PROTOTYPE  EORMS  AND  REPORTS  EROM 
STUDY  REPOSITORY  TEMPLATE  DATABASE 


Main  Menu  Form 


®  Main  Menu  Form  :  Form 


Update  Actigraph  Status 


Update  PDA  Status 


Input  Study  Info 


Input  Research  Team  Info 


Demographic  Survey 
Lark  or  Owl  Survey 


Participant  Information 


Assigned  Actigraphs  Report 


Assigned  PDA  Report 


Actigraph  Sign-Out 


Log  Report 


Assign  Actigraphs 


Assign  PDAs 


Collect  Equipment 


Exit  Database 


Actigraph  Status  Input  Form 


1 S  Actigraph  Status  Form 

^nj 

NAVAL 

^  POSTGRADUATE 

SCHOOL 

► 

Actigraph  ID 

Date  Battery  Changed  |03/27/03 

(mm/dd/yy) 

Timestamp 

3/28/2003  11:34:14  AM 

1 1  Record:  1  ^  1 1 

1  ►  1  ►!  h*|  of  25 
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PDA  Status  Input  Form 


6i  PDA  Status  Input  Form 


^xj 


NAVAL 

POSTGRADUATE 

SCHOOL 


PDA  ID 
ARES  Loaded 


HSILl 


[7 


HanDBase  Loaded 


Log  Database  Loaded  ^ 

Record;  i<  I  ^  1 1  T  ►  I  »i  !►*!  of  19 


Study  Input  Form 


no 


Research  Team  Input  Form 


Participant/Demographic  Input  Form 


III 


Lark  or  Owl  Input  Form 


M  Lark  or  Owl 


^X] 


naval 

POSTGRADUATE 
SCHOOL 


Participant’s  Last  Name; 
Participant’s  ID: 


U 


1.  During  your  last  vacation  week,  how  often  did  you  get  up  laterthan  planned  or  have  difficulty  getting  ready  on 
tie  even  though  you  went  to  bed  at  our  regular  time? 

It  never  happened 
I  It  happened  i-3timesi 
(“  It  happened  more  than  three  times 


When  you  have  no  commitments  the  next  day,  at  what  time  do  you  go  to  bed  compared  with  your  usual  time? 


C'  Seldom  or  never  later 
^  Not  more  than  one  hour  later 
(*  Between  one  and  two  hours  later 
More  than  two  hours  later 


3.  Suppose  that  you  have  decided  to  exercise  twice  a  week  with  a  friend.  The  only  time  your  friend  can  make 
it  is  from  7:00  a.m.to  8:00  a. m.,  twice  a  week.  Assume  you  decide  to  go  at  these  times.  Taking  into  account 
how  you  usually  feel  in  the  morning,  how  would  you  do? 


Record:  H  I  <  IF 


^  Very  well 
(*  Well 
^  Poorly 

C  \ICK-..  Dr,r,KUf 

1  ►  I  ►! \>*\  of  23 


zi 


Assign  Actigraph  Input  Form 


lasMi  -I 


Actigraph  Number 
Participant's  Last  Name  |Smith 
Participant's  ID 


3 


Record;  l<  I  < 


ID 

rr 

— 

Previous  | 

Next  1 

Close  1 

zl 

1  ►  1  H  \>*\ 

of  23 
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Assign  PDA  Input  Form 


Collected  Equipment  and  Data  Checklist  Form 


Collected  Equipment  and  Data  Checklist 


NAVAL 

POSTGRADUATE 

SCHOOL 


► 

Lastname 

IsJiilHB 

J 

ParticipantID 

1  1' 

Consent  Forms  Signed 

R 

Collected  Actigraph 

W 

Collected  PDA 

R? 

Previous  |  Next  |  Close  | 

Record;  i<D||  T  ►Uilt*! 

of  20 
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